Spend five minutes on LinkedIn and you’ll find people announcing AI certifications, agent orchestration projects, MCP implementations, and automation wins. Business analysts are becoming AI engineers. BI developers are becoming data engineers. Operations managers are building autonomous agents.
Apparently, we’re all solution architects now.
We’ve made it nearly free to propose software without making it any cheaper to be wrong about software.
Become the Bear
There’s an old joke about two people running from a bear. One stops to put on running shoes, while the other points out that he’ll never outrun the bear. He doesn’t have to. He just has to outrun the other guy.
AI has created a similar dynamic at work, except people have decided the safest strategy is to become the bear chasing everyone else down.
Employees see the headlines. AI is coming for developers, analysts, customer service reps, accountants, and everyone else with a keyboard. The rational individual response is to get ahead of it.
It’s better to be the person deploying the automation than the person whose job gets automated.
That isn’t cynical. Most people want to improve their work, contribute, and stay employable, and I don’t fault anyone for it. But the incentive to claim technical authority arrives well ahead of the expertise required to exercise it, while AI has made it remarkably easy to look the part.
So everyone is sprinting to become the bear, and almost nobody is asking whether the bear actually knows what it’s running after.
Unqualified people are building production technical solutions. They believe they’re qualified, of course. They just don’t own the consequences when it’s wrong.
The Illusion of Expertise
Generative AI is remarkably good at letting people perform tasks outside their expertise.
Need a Python application? Ask ChatGPT. Want to integrate with a database? Ask Claude. Need an architecture for five autonomous agents that read documents, create purchase orders, calculate invoices, and email customers?
Apparently, there’s a bootcamp for that.
In a few hours, someone with no software engineering experience can produce something that looks like a technical solution. Diagrams, agent definitions, prompts, integrations, working code. The demo is impressive. Some of it works.
But the blind spots are real.
A qualified solutions engineer knows there’s a big difference between producing something that looks like a technical solution and engineering one.
Take the example of an agent built to process repair invoices. It reads the repair documentation, determines the applicable charges, calculates the total, generates a purchase order, and emails the customer. Several manual steps automated away. Think of the productivity gains!
Except billing requires precise, repeatable results. It depends on contractual terms, approved rates, business rules, taxes, authorization limits, and authoritative transactional data.
A large language model is probabilistic. That makes it useful for interpreting ambiguous information, and a pretty terrible choice for independently determining what a customer owes.
Could AI help interpret an unstructured repair document? Absolutely. Could it extract information that a conventional application validates against known business rules? Certainly.
But the financial calculation and the transaction belong in deterministic systems, with controls and validation around them.
That’s before we even think about authentication, authorization, data integrity, audit trails, error handling, transaction management, or what happens when the agent processes the same request twice. Those concerns are frequently overlooked in demonstrations and citizen-developed solutions. They are, of course, a linchpin of operating reliable enterprise software.
The problem isn’t that someone without an engineering background had the idea. The problem starts when a generated design is mistaken for a validated solution architecture, by someone without the background to see what’s missing.
AI produces convincing answers to questions its users may not be qualified to evaluate. More importantly, it doesn’t raise the questions those users don’t know enough to ask.
Some of the Drones Will Get Through
For years, enterprise technology organizations have struggled to manage demand.
Business users identify problems, submit requests, and work through project managers, business analysts, architects, and engineering teams. It isn’t always efficient. Sometimes it’s painfully bureaucratic, and occasionally the process seems designed to ensure nothing ever ships.
But requirements analysis, architectural review, and engineering discipline exist for reasons beyond filling calendars. Resources are finite, systems have dependencies, security matters, and not every business problem needs new software.
AI changes the economics of generating proposals. A business user no longer arrives with a business need. They arrive with a complete AI-generated architecture and instructions for IT to implement it. Instead of describing the outcome they need, they’re presenting a technical solution they believe they’ve already designed.
An AI-generated architecture isn’t a solution. It’s a proposal, and one carrying technical assumptions the business user hasn’t considered and isn’t qualified to consider.
It still needs requirements analysis, feasibility assessment, security review, integration design, and a judgment about whether the proposed technology or solution is the right one.
Now picture hundreds of employees across dozens of departments, each able to generate increasingly sophisticated proposals at no cost. Add vendors selling AI capabilities, consultants promising transformation, and executives hunting for evidence that the company is embracing the technology.
That’s a drone swarm. Every one launched by someone trying to outrun something – career obsolesence, IT friction, or a fear of becoming the hunted. The sum of it all is volume, velocity, and enough surface plausibility that every one of them takes real attention to address.
Most organizations have capable security teams, architects, and engineering leaders who catch the bad ones. But air defense has a throughput limit. As volume climbs, review capacity doesn’t keep pace, and the pressure shows up as shortened reviews, accepted assumptions, and sometimes controls being quietly bypassed.
Enough of them get through.
We’ve seen versions of this before with shadow IT, citizen development, departmental databases, and spreadsheet-driven processes. Technology organizations handled those badly too, and long delivery timelines are one reason they persist.
What’s different now is the reach of what an inexperienced person can build. An Excel macro causes problems. An autonomous agent with access to enterprise databases, customer documents, financial transactions, and email is an entirely different class of blast radius.
A poorly secured integration exposes customer information. An automation corrupts transactional records or creates duplicate financial obligations. An agent acts on incomplete information, makes an unauthorized change, or sends inaccurate communications to customers.
Keep in mind, AI didn’t invent new categories of software failure. These are things we’ve spent decades building engineering practices to mitigate, and AI hasn’t eliminated these practices nor the need for them. AI or no AI, customers still expect the service they’re paying for to be available, do what they’re paying for it to do, and for the bill to be accurate.
Add in ungoverned automation, and the fact it was well-intentioned or “vetted by my AI” means little. Customers get incorrect invoices, service disruptions, or exposure. And the technical teams – not the business unit that used AI to build its solution – then spend weeks reconciling bad data and repairing the damage caused by one person’s solution that was oblivious to another person’s dependency.
I’m not arguing that AI-generated applications are inherently unsafe or that failure is inevitable. Properly engineered AI systems are reliable and valuable. But the key there is “properly engineered”, and non-technical folks will have a gigantic blind spot on what the word “properly” actually means.
And expecting AI to allow business users to magically bypass actual solutions engineering risks producing a substantially higher rate of production failures, reaching customers who never agreed to participate in the experiment.
Expertise Isn’t Optional
AI is a generational shift. I use it every day, and I expect its role in enterprise technology to keep expanding.
I also don’t believe the answer is to restrict AI experimentation to IT departments. That would be counterproductive, and traditional technology organizations have earned their reputation for being slow, territorial, and resistant to change.
But think through the realities for a moment. Who is responsible when a citizen solution calculates something wrong, allows someone to gain access to data they shouldn’t have, or sends a customer invoice incorrectly, perhaps twelve times?
AI doesn’t change who owns those failures, who has to investigate them, or who has to remediate the damage. The business remains accountable for its operational decisions, but the burden of diagnosing and repairing the technical damage still falls on the technical teams.
Which is what the endangered-professions lists get backwards. For the past few years the conversation has been about which technology jobs disappear. Developers, business analysts, project managers, and data engineers have all made the lists. I’ve even wondered about my own career.
I used to think AI would substantially shrink the need for project management and business analysis. I don’t think that anymore.
Administrative project management, status reporting, and routine coordination are still obvious automation candidates. If all you do is schedule meetings and hold standups, your career might indeed be imperiled.
But the experienced business analyst who can separate an actual business requirement from an AI-generated implementation proposal – and can push back on the business user, even saying “no” – gets more valuable as the swarm grows.
Someone has to evaluate the business problem, establish measurable outcomes, challenge assumptions, and decide whether a proposal deserves engineering resources at all. Those are precisely the functions enthusiastic AI adopters mistakenly assume they’ve already performed, as they race to become the bear chasing down everyone else.
The same logic holds further down. Generating code, diagrams, and agent workflows is available to nearly anyone now. Understanding how enterprise systems interact, how data should be managed, where security boundaries belong, and what happens when an integration fails, however, is a different discipline. Experienced engineers know the apparent simplicity of a business request usually conceals real technical complexity. They know the difference between transactional and analytical workloads, why data integrity matters, and why an application that works in a demonstration is still unsuitable for production.
Most of all, experienced solutions architects know when AI is the right tool and when it isn’t.
I don’t need a large language model to calculate an invoice total. I need a deterministic calculation using validated inputs and approved business rules. I might use an LLM to read the document those inputs came from, but that doesn’t mean the model should own the financial transaction. This is just boring engineering, not AI resistance.
AI can help someone without a technical background produce the appearance of an engineered solution, without giving them the expertise to evaluate what they’ve built. Perhaps more importantly, an AI-generated solution doesn’t move responsibility for its outcomes onto the desk of the non-technical person who generated it.
And where organizations fail to recognize this asymmetry, the probability of a drone swarm causing damage increases. Career incentives, executive enthusiasm, and the ease of AI-generated development are pushing people across professional boundaries, encouraging them to become the bear in areas where they lack both the expertise to make informed technical decisions and the responsibility for their consequences.
The greatest enterprise AI risk isn’t that AI replaces experienced engineers. It’s that organizations decide they no longer need that expertise while firing off more and more drones that depend on it.
Eventually the distinction between a convincing demonstration and a reliable production system becomes impossible to ignore, when things like incorrect invoices, corrupted transactional data, or angry customers inevitably show up.
By then, the bootcamp is over.
–Scott
