For many organizations, AI is proving easy to pilot but difficult to scale. Pilots often look inexpensive because they run on narrow datasets with a handful of users, explains Ben Schein, chief AI and analytics officer at cloud software company Domo. “But the cost lives in deployment, the moment you connect that capability to real workflows and the systems of record behind them,” he says. “That’s when the real bill appears.” So CIOs must always budget for the gap between when it works in a demo and when it produces governed and durable value.

Organizations can easily get caught out because they run pilots as a technology experiment instead of a business initiative, he adds. “The interesting question is never whether AI can do the thing in a demo,” he says. “It’s whether it should run in this process, and whether it survives contact with production.”

There’s also a lot of pressure on IT teams to be doing something with AI simply because everyone else is, says Naren Gangavarapu, chief transformation and AI officer at Australian Cruise Group.

He calls it AI theater because there’s a big show around AI even though there aren’t that many successful applications of the technology in production environments.

AI costs out of control

According to John D’Emic, CTO at AI observability platform Revenium, one of the big traps when running a pilot is failing to anticipate how quickly consumption can spiral as adoption grows. “As an example from our own engineering org, back in May, a developer opened an AI coding session on his laptop, and it stayed open for four days,” he says. “By the time it closed, it had run 4,819 calls and cost us $3,762. We didn’t budget for this, and no alert fired. But that one session cost more than a lot of teams spend on their entire monthly AI tooling.”

While this showcases how a developer can make a costly error, Dmitriy Anderson, CIO and digital and social commerce leader at home and gardening retailer Leroy Merlin South Africa, believes the pilot trap frequently happens when employees with little or no software development experience vibe code applications. “It doesn’t matter if you can create something in 15 or 20 minutes if the result is AI slop,” he says. “Think dirty code, no consideration for safety, security, and possible data exposure.” In most cases, these pilots are developed with one of the frontier apps, and someone probably used their personal AI subscription, so the costs are negligible, he adds. But if you have a company of several thousand people, and you now want to roll this tool out more broadly, that’s where costs can get out of control.

This scenario is only exacerbated by the introduction of agentic AI, D’Emic adds. “Agents don’t spend money at human speed,” he says. “In the old cloud days, an engineer could spin up infrastructure in minutes and finance might not see the bill for a month, which was painful but recoverable. Agents, though, call APIs around the clock without waiting on anyone’s approval.”

Mind the trap

While cost is a big factor in the AI pilot trap, it should be treated as a symptom of a bigger problem, says Schein. The underlying issue is governance and observability. “An autonomous workflow can fan out into more queries, API calls, and model invocations than anyone scoped,” he says. “So if you can’t see what it’s doing, and spend compounds quietly, you only find out once the invoice arrives.”

In a recent LinkedIn post, Anderson outlined how in just six weeks he built a platform for a fraction of the sticker cost using three AI models orchestrated together. The traditional estimate to build the same tool would have required 2,472 engineering hours from a team, and was expected to take around nine months. “I went through the proper engineering steps and planning, and made sure the application passed a series of cybersecurity frameworks,” he says. “The purpose of this exercise was to showcase that AI can still speed up the process even if you take the time to work through the necessary steps. You can build with AI rigorously and securely.”

So to turn AI experiments into enterprise value, every AI interaction must be attributable: who triggered it, against what data, on which model, and at what cost, Schein says. For each workload, be sure to ask how often it runs, which model tier the job actually needs, and what triggers it, human or automatic. “A frontier model on an automatic trigger and a small model called on demand are completely different cost curves for the same task,” Schein adds.

For Anderson, it’s helpful to use AI to highlight potential gaps, assumptions, or blind spots in your ideas early on. “When you start building an idea, ask the agent to interview you,” he says. “It will go through every phase and ask questions about the important facets of the process, from scalability and budget to deployment options. You can even make AI write a prompt for itself, because it knows its capabilities and quirks better than you ever will. It’s called meta prompting.”

Anil Inamdar, global head of data services for the Instaclustr BU at NetApp, suggests CIOs cost out the whole program, not just the demo. “Generally, the model itself is the cheapest part of the program,” he says. For him, it’s important to have security and governance people in the scoping meeting, not the launch meeting.

He believes the pilot trap is also, or perhaps mostly, a sequencing trap. “A lot of teams are wired to build first and ask permission later, only to discover months down the line they can’t pass a security review or data privacy audit without a painful and costly rebuild. It’s also valuable to define what failure looks like before you define success.

“Pilots tend to die because of no result, which isn’t the same as a bad result,” Inamdar says. “Emphasize to the deployment team on day one that if a target result by a certain month isn’t seen, we shut it down. Otherwise, you’re funding a zombie pilot because everyone’s invested and no one wants to be the one to call it out.”

A few weeks ago, I found myself in yet another demonstration of an application that was not built by my team or my technology partners. The demonstration lasted 20 minutes and by the end of it, everyone in the room agreed that the application solved a genuine business problem. There was genuine admiration for the business team’s initiative, a few questions about future enhancements and the inevitable congratulations that accompany any successful AI story. Then somebody asked, almost casually, “Can IT roll this out quickly across the organization?” It was a short question that unfortunately needed a long and unpopular answer.

I am sure almost every technology leader reading this has experienced some variation of this moment. The application itself is rarely the problem. In fact, many of the applications I’ve seen over the past year have been remarkably good. What concerns me most is something rather different. Somewhere along the way, we’ve started confusing the act of building software with the responsibility of running it. AI has dramatically reduced the effort required to create an application, but it has done little to reduce the effort required to own and operate one sustainably.

That distinction may sound subtle, but it is not. I think this might be one of the most significant leadership challenges of the AI era.

The success nobody planned for

None of this should come as a surprise. For years, we have been telling the business to become more digitally savvy. We invested in low-code platforms and citizen development initiatives, organised hackathons and innovation challenges, and repeatedly argued that technology shouldn’t become a bottleneck to solving business problems. The rapid emergence of AI coding assistants has simply completed that journey. Today, anybody from finance, marketing or operations can turn an idea into a working application faster than most technology teams can schedule a requirements workshop.

Personally, I think that’s amazing. Some of the most interesting ideas I’ve seen this year didn’t emerge from tech teams. They came from people who understood the business problem intimately and no longer needed permission to begin experimenting. That’s a future I would much rather embrace than resist.

The problem isn’t that business teams are building software. The problem is that successful prototypes have a habit of raising enterprise expectations quickly. Yesterday it was a departmental experiment. Today it is being demonstrated to the executive committee. Tomorrow someone is asking why the rest of the organization isn’t using it. The application hasn’t changed; the expectation has. In many ways, this feels like the next evolution of what we’ve traditionally called shadow IT. The difference is that these applications are often better engineered, solve genuine business problems and, ironically, are being built with the very experimentation that technology leaders have spent years encouraging.

The thin line between building and running

This is the point where discussions between business leaders and technology leaders begin to fall apart. The business sees an application that works. Technology sees an application that now needs to survive outside the protected environment in which it was created. Those are fundamentally different things.

A prototype rarely worries about things like identity management, resilience, audit trails, backup policies, API versioning, support models or regulatory obligations because none of those questions matter while an idea is still being tested. However, they become important once the organization decides that the application has graduated from an experiment to an enterprise capability. In many ways, this is the point where software engineering gives way to software operations—a discipline that organizations like Google have spent years formalising through practices such as Site Reliability Engineering (SRE).

This is the point where many organizations are beginning to underestimate the challenge. AI has democratised software development. It has not democratised enterprise operations. Running software is an entirely different discipline. It is less visible, less celebrated and considerably less exciting than building it, but it is also the reason enterprise technology exists. Every application that enters production quietly accumulates obligations. Someone has to secure it. Someone has to integrate it. Someone has to monitor it, patch it, support it and explain it to an auditor. Eventually, someone has to retire it. None of those responsibilities disappear simply because the first version happened to be created in forty-eight hours using an AI.

Walking the tightrope

The temptation for technology functions is to respond in one of two ways. The first is to become the organization’s brake pedal. Every application must now navigate governance committees, architecture reviews, security assessments and operational checklists before it is allowed anywhere near production. The enterprise is undoubtedly safer, but enthusiasm evaporates quickly when innovation feels like it needs intricate planning and convoluted permissions.

The second temptation is more subtle, and in many ways more dangerous. We become so determined not to discourage innovation that every successful prototype quietly becomes another production application. We congratulate ourselves on enabling the business while gradually accumulating a software estate that nobody really owns or understands. Six months later, the original creator has moved to another project, the AI prompts have disappeared, users have doubled, integrations have multiplied and suddenly the technology team is required to support something it neither designed nor approved.

Neither extreme is sustainable.

This, I suspect, is the balancing act that leadership will increasingly be judged on. Not whether we can prevent people from building software—that battle has already been lost, and rightly so—but whether we can encourage experimentation without allowing enthusiasm to become tomorrow’s operational burden.

I’ve discovered that the tone of these conversations changes entirely if we begin with curiosity instead of governance. Rather than asking why technology wasn’t involved earlier, we now ask what problem the team was trying to solve. It sounds like a small change, but it transforms the discussion. People become far more willing to talk about security, resilience and operational ownership once they know those questions are intended to preserve what they’ve built rather than prevent it from succeeding.

Stewardship, not gatekeeping

Interestingly, this isn’t simply a challenge that individual technology leaders are experiencing. Recent research points in the same direction. The 2025 DORA State of AI-assisted Software Development report concludes that AI acts primarily as an amplifier. It magnifies the strengths of organizations with mature engineering practices and exposes the weaknesses of those without them. In other words, the greatest returns from AI don’t come from the coding tools themselves, but from the quality of the underlying engineering and operational system.

Gartner arrives at a similar conclusion from a different perspective. In its analysis of enterprise AI coding agents, the firm argues that the market is rapidly evolving beyond developer productivity towards operational excellence and enterprise readiness. As organizations begin operationalising AI-generated software at scale, governance, operational ownership and long-term lifecycle considerations become just as important as the tools themselves.

None of this should really surprise us. We’ve spent the last couple of years asking whether AI can help us build software faster. That question has largely been answered. The more interesting question now is whether organizations are prepared for the consequences of making software creation almost frictionless. Every successful application creates an obligation that lasts far longer than the weekend it took to build.

Technology leaders have traditionally thought about technical debt as ageing platforms, deferred upgrades, architectural compromises and code that has outlived its original design. CIO.com has written extensively about the long-term business impact of technical debt. I believe AI is quietly introducing another form of debt that deserves equal attention—operational debt.

Operational debt begins the moment an application is promoted from a successful prototype to a business-critical service without a clearly defined operating model. Every application that is enthusiastically pushed into production quietly becomes another long-term obligation. It needs monitoring, support, ownership, governance, funding, documentation and, eventually, retirement. Unlike technical debt, operational debt is rarely visible until something fails, an audit raises uncomfortable questions or the person who originally built the application has long since moved on.

The role of leaders may no longer be to decide who gets to write software. AI has already democratised that capability. Our responsibility is something altogether more nuanced. We have to preserve the excitement, curiosity and initiative that AI has unlocked across the business while ensuring that the enterprise remains secure, resilient and supportable. Push too hard and we become the one that quietly kills innovation. Push too little and we inherit an estate of applications that nobody is truly prepared to operate.

This isn’t a governance problem. It’s a leadership one…and I suspect it may well become one of the most defining responsibilities of enterprise technology for years to come.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?