Every time an AI agent causes a catastrophe, the conversation goes to the same place. What did the AI do wrong? Can these systems be trusted? How do we stop it happening again?

Those are the wrong questions. And the wrong questions lead to the wrong fixes.

The better question, the one that actually leads somewhere useful, is this: who built the environment where it could happen?

 

Three Incidents. The Same Root Cause.

PocketOS, April 2026. An AI coding agent found an unscoped API token sitting in an unrelated file. It used that token to delete a storage volume on Railway, their infrastructure provider. It did not check whether the volume was shared across environments. It was. Production database and every backup, gone in nine seconds.

Replit, July 2025. An AI agent deleted over a thousand executive records during an explicit code freeze. Nothing stopped it because nothing had been configured to stop it.

Amazon Kiro, December 2025. An AI agent inherited a senior engineer’s elevated permissions, the kind that would normally require two people to sign off on a destructive action. It deleted and recreated an entire cloud environment. Thirteen-hour outage.

In every case, the agent did something it was technically permitted to do. Not something it was asked to do. Something it could do, because the architecture said it could.

That is not an AI failure. That is a design failure.

 

We Are Asking Questions About the Wrong Moment

The instinct to interrogate the AI is understandable. These systems are new, they are powerful, and when they cause damage the natural response is to look at the machine.

But that framing lets the actual problem off the hook. In each of these incidents, someone made decisions about credential storage, access scopes, permission inheritance, and whether destructive actions should require human confirmation before execution. Those decisions created the conditions. The agent had the speed and autonomy to find the gap before anyone noticed it was there.

“The incident is never the origin.” Every one of these failures has a human design decision sitting upstream of it.

 

CIOs and CTOs Own This

This is where the conversation needs to land, and where it rarely does.

CIOs and CTOs set access models. They decide, or delegate the decision about, what credentials AI agents can reach, what permissions they inherit, and whether irreversible actions require a human confirmation step. These are not AI product decisions. They are infrastructure and governance decisions of the kind that technology leadership has been making for years.

The least-privilege principle has been a security standard since the 1970s. Every process should have only the access it needs and nothing more. We have applied it carefully to service accounts, automated pipelines, and human users for decades. We are not applying it with the same rigour to AI agents. The gap is showing up in production.

 

Three Questions That Determine Your Exposure

If you are deploying AI agents and cannot answer these clearly, you have a governance problem. It is a matter of when, not if.

• Are your AI agents operating on minimum permissions? Or are they inheriting ambient credentials that happen to be accessible in the environment? Unscoped tokens stored in accessible files are a credential hygiene problem. AI agents now have the speed to exploit them in ways a human operator simply would not.
• Do irreversible actions require human confirmation before execution? Not a log entry after the fact. A genuine gate, before the command runs. Deletion, overwrites, production deployments. These should not be single-step autonomous operations regardless of how capable the agent is.
• What is the blast radius? Before any AI agent is deployed, you should be able to answer: what is the worst thing this agent could do with the access it currently has? If that question gives you pause, the deployment is not ready.

These are not new questions. They are the questions we have always asked about automated systems. The difference is that AI agents are faster, more capable of creative problem-solving, and more likely to find an unintended path that nobody anticipated during design.

 

Governance Does Not Require a New Platform

Much of the current enterprise AI governance conversation focuses on model behaviour: hallucination, bias, output quality. Those are real concerns. They are not the ones that will delete your production database.

The vendors now selling AI governance infrastructure are not wrong about the problem. But executives should make their own assessment of what their environment actually needs. “Governance does not require a new platform. It requires applying principles you already know to systems you are now deploying.” When vendors know the renewal depends on value delivered rather than fear managed, the conversation changes quickly.

 

The Agent Did Not Design the Environment

The uncomfortable fact sitting underneath every one of these incidents is that the AI agents involved were, in a narrow technical sense, doing their jobs. They identified a problem and attempted to fix it. They used the access they had. They executed what was permitted.

The humans who made the architectural decisions upstream of those moments are the ones who need to answer for the outcomes.

You cannot fix an architecture problem by retraining the model.