Organizations are racing to deploy AI, but warning signs are accumulating. Earlier this year, an internal AI agent gave an engineer instructions that exposed sensitive user and company data for two hours. Around the same time, a large online retailer issued a 90-day safety reset after its AI assistant contributed to an incident that involved nearly 120,000 lost orders. And in the spring, an AI agent deleted a company’s production database and its volume-level backups in nine seconds.

Over recent years, recurring events like these, among others, expose a widening gap between what AI can do and what organizations can safely control.

“A year ago, most conversations were about accelerating AI adoption as fast as possible,” says Sandeep Johri, CEO at application security platform Checkmarx. “Today, boards ask tougher questions. Speed and governance have to move together now.”

As Johri points out, the new bottleneck is the organization’s ability to govern AI. According to IBM’s 2026 Tech Leader Study, 77% of organizations admit their governance is failing to keep pace with AI. And, among IT executives, 70% say business teams are deploying tech faster than it can be tracked.

The use of agentic AI only widens the gap. About 80% of the organizations surveyed say they lack mature capabilities for it, according to Deloitte. That includes clear boundaries for agents, real-time monitoring systems, and audit trails that can capture the entire chain of actions.

CIOs need to operate in this paradigm to address two competing demands: accelerate AI adoption to boost productivity and outsmart competitors, and assure boards that all sensitive data is protected and AI only does what it’s supposed to do.

“I don’t think you can separate the two,” says Sahil Sanghvi, VP of AI engineering in the chief technology office at Booz Allen Hamilton.

Innovating while managing risks

At first glance, AI-generated code can look good and even pass initial testing. A thorough review, however, can shed light on multiple issues. This is something Ha Hoang, CIO at data protection platform Commvault, witnessed firsthand.

In one case, her team found the AI had taken a shortcut. It bypassed the company’s authentication process in favor of a simplified implementation, which lacked established access controls. “Without those checkpoints, it could’ve made its way much further,” she says.

When companies discover major issues, they should immediately pause deployment. But many problems aren’t obvious. “AI-driven risks often remain hidden, and moving too quickly only makes those silent failures harder to detect,” says Omer Cohen, CISO at customer identity and authentication service Descope.

But to strictly move slowly everywhere isn’t an option either. The idea is to identify where speed creates value, and where the potential consequences call for caution, and then build necessary guardrails case by case.

For Bob Leek, CIO at Clark County, Nevada, that means making governance and compliance part of the design, not a final check before deployment. “We’ll go slow to go far instead of going fast and creating risks,” he says.

The biggest challenge is organizational, not technical

In many cases, AI deployment is less a technology problem than a people problem. When deciding what to automate inside an organization and how to do it, the real challenge is understanding how work actually gets done. And usually there are many invisible, undocumented processes that influence it.

Employees in HR, finance, procurement, legal, or operations rely on exceptions every day. They have workarounds and make judgment calls to keep the organization running. These tweaks are simply part of the job, so they rarely think about them or include them in official process documentation.

These elusive workflows can’t be mapped simply by considering how things are supposed to work. Leaders must closely observe how employees actually do their jobs.

“Frontline teams understand the exceptions, escalation paths, and context that rarely appear in a process map,” says Leek. “We bring those teams into the design process, mapping the handoffs and non-standard cases.”

Cohen agrees. “Invisible threads are often fragments of context residing in an individual’s mind rather than a database,” he says. For instance, an analyst may know that a client’s login spike is harmless because it’s scheduled during weekly testing. “Unless this tribal knowledge is codified as a formal governance artifact via runbooks, threat models, or decision logs, no AI will naturally possess it,” he adds.

But simply asking employees how they work isn’t enough, adds Amitkumar Rathi, chief product and technology officer at hybrid infrastructure observability platform Virtana. The best approach is to run shadow sessions, in which someone in tech actually witnesses how the work is done. “We sit next to them during live incidents and ask, for instance, why did you look at that dashboard and not this one; why escalate now and not 10 minutes ago; what told you this was the same issue as last month’s incident and not a new one?” he says.

Of course, mapping informal processes takes time and discipline, and there shouldn’t be any tempting shortcuts. “The organizations that get this right treat AI as a collaborator in their existing workflows, not a replacement,” says Vijay Jegan, chief AI transformation officer at enterprise customer retention platform Gainsight. “Success requires a hybrid of deep business acumen within a department and the technical maturity to understand the inherent risks of modern AI tools.”

But not all tribal knowledge can or should be documented. “The goal should be to architect AI to augment this human foundation, rather than attempt to replace it entirely,” adds Cohen.

Where should humans stay in the loop

Giving AI a larger role makes human judgment more important, not less. “Humans should stay in the loop in every decision, but not every part of the process,” says Leek. “The urgency to innovate doesn’t change that fundamental responsibility.”

CIOs can decide where people should remain involved by weighing the value of human judgment and the risk of leaving the task entirely to AI. Tasks that score highly on both should remain firmly in human hands. “The higher the risk, the more human oversight is required,” Jegan says.

Sanghvi also factors in human consequences of potential AI mistakes. “When you deal with a decision that could materially affect a person, a mission, or an organization, that’s where you want clear human authority to intervene or override the system,” he says. “As AI becomes more agentic and starts taking actions rather than just making recommendations, being clear about those boundaries becomes even more important.”

Meanwhile, Cohen draws the line at AI-powered decisions that can’t easily be undone. “Human intervention remains non-negotiable at any juncture where a decision becomes irreversible or traverses a critical trust boundary,” he says.

At the other end of the spectrum, routine, low-risk work can be left to the machine. “Organizations may trust agents to autonomously handle narrow, repeatable tasks,” says Hoang, adding, though, that even advanced agents can misinterpret context or take unintended actions at scale.

“The future isn’t blind trust but measurable trust built on transparency and control,” she says.

Governance doesn’t end at launch

Before an AI initiative becomes a major commitment, Leek recommends CIOs ask if the project supports the organization’s strategic priorities, if IT can support it, and does the business department have the capability and appetite to change?

“This framework helps prevent initiatives from becoming solutions in search of a problem,” he says. It also helps CIOs start with lower-risk projects, test what works, and strengthen governance before applying AI in more sensitive areas of the organization.

Clark County took that approach with its first AI deployment for special-event permitting. Its AI tool guides promoter through forms, identifies the permits needed, and connects them with a county analyst. But starting with a lower-risk project doesn’t mean the governance work ends at launch. Governance should be a continuous conversation rather than a checkpoint, says Sanghvi, since data changes and models evolve.

Hoang agrees. “If your governance system relies on quarterly reviews, you’re already behind,” she says.

Over the course of his career, Jim Reavis has seen cloud and cloud security evolve, and it’s come a long way since being a niche technology in the early 2000s. Now it’s dominant in terms of being the IT foundation, he says, but while the tech is strong, the operating models is where things get messy. Cloud, security, and third-party risk teams look at different parts of the problem, of course, but challenges remain.

“Operational technology worries me a great deal,” he says. “A lot of those systems are isolated and not kept up to date. If we don’t modernize them, we’re going to have huge problems. In a lot of cases, things fall between the cracks and that’s where hackers like to exist.”

So much of what’s around the models is where cybersecurity has responsibility, rather than the provider covering everything. “Data, identity, and applications are shared responsibility areas, and in many cases, the tenant carries most of the control burden,” Reavis says. “If you use a hyperscaler, you may still have about 80% of the responsibility for the controls around what you build.”

And when it comes to AI, the model isn’t the whole problem. What matters is the context around it, the goals it’s given, and the oversight put in place, he says. “We need to think carefully about the harnesses we put around AI and the systems we use,” he adds.

The responsibility model, therefore, is a recurring issue in cloud security breaches tied to misconfiguration and accountability gaps, and some enterprises still aren’t clear about where responsibility begins and ends. “We spent a lot of time on a shared security responsibility model, but when this first started to gain popularity, there were a lot of organizations or SaaS providers you could work with who’d say it’s in the cloud, it’s at Amazon,” he says. “Look at their certifications and SOC2 and how they comply because they’re covering everything.” But when you look at the actual applications, data, and identity, he adds, there’s so much that’s shared responsibility, and the customer’s responsibility.

So how do we make sure information is encrypted properly so it doesn’t become a tenant issue? “There’s still a bit to do, and we think about this not only from whether it’s SaaS, infrastructure, or a particular provider, but at what level is it at the physical, network, or audit level,” he says. “And even from a role-based perspective, what’s the role of internal risk and role of providers?”

Reavis gives further detail about how AI adoption exposes weaknesses in identity, trust, and risk management, and the long-term implications of increasingly interconnected cloud ecosystems. Watch the full video below for more insights, and be sure to subscribe to the monthly Center Stage newsletter by clicking here.

On cloud risk management: When we had the Chat GPT moment, we knew it because AI had been around for a while, but that was a cloud delivered version of AI to the masses, so we saw this going to evolve and you could see it combining in many important areas.

But what we’ve learned is, because this is an interesting predictive rather than deterministic technology, we’re living in a world of two exponentials, and you’re seeing model capabilities growing so quickly. There’s this feeling from a security perspective that we have to look to the model itself and fix every hallucination and everything else when that’s built into how it works. It’s actually working as intended. So that’s a new lesson. Models are going to get more powerful, but it’s so much of what’s around the models where cybersecurity has responsibility, and we don’t rely on frontier model companies or using open-weight models. Rather, what’s the context, oversight, and information we’re providing them, what do we do in terms of goals we give them, and what are the harnesses we put around AI and the models we deal with?

These are going to be the big areas to think about, but we have to understand the parts we can control. We’ve got to think carefully about the harnesses, transparency, and using supply chain shared responsibility. SaaS and cloud providers are all AI enabled now. You’re not using any software of any significance that isn’t using AI to some degree.

On AI identity, trust, and control: One of the areas that we’ve championed is zero trust as a philosophy. It was initially more of a networking type of approach at the network layer, or an idea that you use identity to understand network access. But it’s evolved more to an idea that anything can be breached, so you assume that. Then you think about how to make systems resilient, and build up confidence and protection.

So zero trust tells us that with human identity, we can ask what our digital identity is, and now we’re in a very interesting area for identity management and associating that with agents and AI systems. People might have just one view of it, but agents are as diverse as humans. So we think about different identities and least privilege, and how to prevent them from escalating privileges. We need to introduce new concepts like least autonomy, and think about an agent that has certain tasks and use identity to make sure the actions it takes are within a defined scope. Because while we’ll see a lot of security incidents with AI, proportionately we’ll see more misconfiguration and bad things that happen because of broken processes. And the AI system just deletes things because it thought that’s what it’s supposed to do.

So it’s important to make strides in how we think about identity and agents, and the idea of digital workers. How do we manage and treat those? If we think about them too much in either one of those realms, we’re going to fail. So we have to understand what’s the right blend. It’s a new area and very exciting.

On risk and legacy systems: When I think about operational technology, sometimes systems are isolated and not kept up to date. That concerns me a great deal. We’re going to have huge problems there. We have concerns about existential risks, where people don’t want to use the latest technologies and be aggressive adopters of AI. I think that’s going to create real scale issues with organizations.

So we have to understand where we are, where we’re going, and have a vision that serves something between human and technology, maybe a hybrid, but we’ve got to make our peace with it and understand the appropriate harnesses and direction where humans should always be in the loop with control. But it’s appearing in some new areas of cybersecurity where we haven’t traditionally thought about. Software development looks very different now than it did 12 months ago, and 12 months from now, cybersecurity is going to be really different, too.

On cloud security and implementation: Cloud security is cybersecurity for all intents and purposes. We have so much tooling and technology that’s really good, but there’s a lot of inconsistencies with the operating models organizations have. Even way back with CSA and NIST defining this, it was clear that SaaS was a layer on top of infrastructure as a service. But we diverged, and you see in a lot of enterprises there’s diffused ownership where you have cloud and security teams, and then you have third-party risk that deals with the SaaS team. Then there are inconsistencies in how risks are managed, so internal development and expectations from our partners can really diverge. They have a lot of regulations to deal with, so it creates vetting and investment challenges while striving for consistent models.

Some security teams might still use older checklists to talk to their cloud teams, but scaling with new tech becomes an issue if you’re not thinking about operations. It ends up being a human and a structure problem that makes it harder to take advantage of all the great technology that’s out there.

Businesses received a staggering amount of cyberattacks in June, according to Check Point, showing a rise of 20% over the previous 12 months. The breakout of AI agents from OpenAI in July to hack into the Hugging Face website, and subsequent similar events from Anthropic and Meta, indicate agentic-powered attacks will explode over the coming year.

Currently, malicious hackers have the advantage because publicly released frontier models from the US incorporate guardrails that can’t distinguish between malicious or defensive activities. As a consequence, these models default to a refusal to get involved. Hugging Face discovered this the hard way when they attempted to utilize a model to defend against the OpenAI intrusion. Their solution was to adapt a Chinese open weight model to analyze the 17,000 attack logs, find the vulnerability, and contain the intrusion.

With incidents like these happening more often, an arms race has begun with AI being both the problem and the solution.

Strength in numbers

While single agents generally perform more efficiently for well-defined tasks, research from Stanford University indicates swarms are more effective in messy scenarios with noisy data, which are more typical of unpredictable, intrusion attacks. The increased token usage by swarms raises costs, but increasingly efficient open weight models are rapidly lowering these barriers.

In the Hugging Face example, the agents worked together as a team leaving messages for each other on a message board they improvised. They shared newly found vulnerabilities, exchanged tools, and even developed conventions to address one another and to avoid overwriting each other’s work. While this may seem sinister, they were only following their designated purpose: to achieve a goal without regard to any collateral damage. We can expect bad actors to harness the power of agentic swarms through fine-tuning open weight models, and creating agents that progressively learn from their experiences.

Modern warfare has been transformed over the last four years, too, through the deployment of drones by Ukraine to defend against Russian attacks. Military strategies and the deployment of armament budgets around the world are shifting to focus on new technologies, and approaches and enterprises are now facing a similar challenge from the hostile use of agentic AI.

The drawbridge is down

As enterprises build out their own agentic systems to handle ecommerce, customer service, and marketing activities, this presents new attack surfaces for antagonistic efforts. April 2026 research from Trend Micro found almost 1,500 MCP servers directly exposed to the internet had no authentication or encryption, a rise of 200% from nine months earlier. This included 70 hosts offering direct SQL execution, and servers holding medical records.

The automation of business processes and the reduction of humans from decision making chains open up new vulnerabilities for agents with malicious intent. Arkose Labs’ 2026 agentic AI survey of 300 enterprise leaders found 97% expected an AI agent security incident within the next 12 months.

Social engineering

While agents have demonstrated their ability to break through security systems, they’re also capable of targeting humans to achieve their objectives. Recent research from Verizon indicates that 62% of successful breaches involve a human element, with phone-based attacks 40% more successful than email-based ones. In August, for instance, scammers using an AI-generated deep fake of Australian Prime Minister Anthony Albanese’s voice were able to scam investors out of $5.3 million.

If agents can break out of digital sandboxes, and generate convincing fake videos and audio, then they’re certainly capable of making basic phone calls. In July, during testing of frontier models, the UK AI Security Institute discovered an agent tried to insert malicious code into an open-source project. Attempting to get the code approved, the agent created fake online identities using them to persuade the project’s maintainer to sign it off. “This is the first time we’ve seen risks around autonomy and deception manifest this clearly without specific prompting in the real-world,” the Institute put in a write-up of the incident.

Fight AI with AI

So attackers currently have the upper hand in this escalating arms race. They have access to agents that can work around the clock, constantly probing, learning, and sharing their knowledge with other agents. They’ll only get better at this and learn ways to stay ahead of defensive systems. International agreements to delay or restrict the capabilities of frontier models won’t stop hostile actors motivated by money or rogue states pursuing other objectives. Developers and security vendors need access to the latest frontier models unfettered by restrictive guardrails if we’re to stand any chance of defending against the coming tsunami of attacks.

We can learn a lesson from recent history on this front. In 1992, the US restricted exported software to weak 40-bit encryption, citing security concerns going back to the cold war. While the US allowed stronger encryption internally, the result was weakened security for everyone as hostile antagonists were able to disrupt global supply chains that incorporated less secure software. Despite lifting the ban in 1999, embedded software containing 40-bit encryption continued to cause problems for many years across multiple countries, including the US.

Without rapid action, we may look back fondly to the world before July 2026 as a golden age for cybersecurity, a relative age of innocence.