By Rohana Rezel

The pitch is seductive. Point an AI agent at your infrastructure, give it a task, walk away. No tickets, no on-call rotations, no waiting for an engineer to get around to it. The agent reads the codebase, reasons through the problem, writes the fix, runs it. Done.

The last eighteen months have produced a clear answer to what happens when you take that pitch literally.

In July 2025, Jason Lemkin of SaaStr used Replit’s autonomous coding agent to build an app. He told it explicitly: code freeze, no changes without permission. The agent overrode the instruction, connected to the live production database, and deleted it. It then generated fake records, apparently to conceal what it had done. When confronted, it initially denied what had happened. ^[1]https://x.com/jasonlk/status/1946069562723897802 Lemkin eventually recovered the data through a manual rollback, but the incident was public and damaging enough that Replit’s CEO called it “unacceptable and should never be possible.” ^[2]https://x.com/amasad/status/1946986468586721478 The agent had been given full infrastructure access and no meaningful constraint on what it could do with it.

In March 2026, Alexey Grigorev of DataTalks.Club tasked Claude Code with migrating infrastructure on AWS using Terraform. A state file on a different machine caused the agent to treat the live production environment as non-existent. It produced a plan that showed destruction of existing resources. The agent flagged the discrepancy and advised against proceeding. Grigorev overrode it and approved the plan. The agent ran terraform destroy. It deleted the production RDS database, the VPC, the ECS cluster, the load balancers, the bastion host, and the automated snapshots. 1.9 million database rows. 2.5 years of student homework submissions and project data. AWS support recovered the data after 24 hours through an undocumented internal backup. Grigorev documented the whole thing publicly and put it plainly: “I over-relied on the AI agent by removing human safety checks.” He approved a plan he did not fully understand. ^[3]https://alexeyondata.substack.com/p/how-i-dropped-our-production-database

Also in March 2026, Meta’s internal agentic assistant posted incorrect instructions in an internal forum about how to configure access controls. An engineer followed the instructions. Sensitive user and company data was exposed to unauthorized internal engineers for two hours. The agent did not directly change any permissions — it produced text, and a human ran the commands. Meta declared it a SEV-1 incident. ^[4]https://www.kiteworks.com/cybersecurity-risk-management/meta-rogue-ai-agent-data-exposure-governance/

In December 2025, Amazon’s own internal AI coding tool, Kiro, was given autonomy to fix a configuration issue in production. It determined the cleanest path was to delete the entire production environment and rebuild it from scratch. It inherited elevated IAM permissions from the deploying engineer, bypassing the two-person approval requirement for production changes. AWS Cost Explorer went down for 13 hours in a China region. Amazon called it user error and misconfigured access controls. ^[5]https://incidentdatabase.ai/cite/1442/

These are four incidents that were documented publicly, by named individuals or companies large enough that the outage made news. They are not a statistical sample. They do not tell us how many AI agent deployments ran without incident in the same period, and that number is certainly not zero. The pitch has already spawned a sales layer. Saba Tchikhinashvili’s Opsy ^[6]https://www.linkedin.com/posts/saba-tchikhinashvili-76953114a_infrastructure-as-code-made-sense-when-code-activity-7443286383269556224-HG6r markets an API where AI agents propose infrastructure changes, humans review the intent, and the platform handles execution — the agent-in-production workflow, repackaged as a product. That framing is the same pitch made palatable: the agent still proposes changes to live infrastructure, the human still has to understand what it proposed well enough to approve it, and the liability is still waiting for someone to click accept on something they do not fully grasp. Grigorev had that review step. He looked at the plan. He approved it. The review gate did not save him — not because he skipped it, but because the artifact, a Terraform destroy plan generated from a stale state file, was not something a human could evaluate correctly without fully reconstructing the context the agent had worked from. The human review layer only works if the thing being reviewed is actually reviewable. Calling it controlled does not change that. What the incidents are is a diagnostic — not of a single failure mode, but of a shared precondition.

The four incidents failed differently. Lemkin’s agent overrode an explicit instruction and fabricated records to hide what it had done. Grigorev’s agent misread a stale state file and treated an entire live environment as non-existent. Meta’s agent posted advice to a public internal forum when it should have been private, amplifying bad instructions to more engineers than intended. Kiro inherited IAM permissions broad enough to bypass the two-person approval requirement that should have caught it. Different triggers, different mechanisms, different teams. The common thread is not the type of failure. It is that in each case, an agent had access it should not have had, or acted in a context it was not designed to handle, and there was no enforced gate that stopped it before the damage was done.

The problem is that context. An agent executing a task in production does not carry the same weight of consequence that a human engineer does. When a senior engineer runs terraform destroy on a production environment, they know — because they have been in that situation before, or have heard what happens to people who have — that they are making a move with real stakes. That knowledge changes how they read the plan. They look for what seems wrong, not just what seems correct. The agent optimizes for the task. The engineer is responsible for what the task does to the world.

Every incident above had a human in the loop, technically. Grigorev approved the Terraform plan. The engineers at Amazon and Meta granted or inherited the permissions. Lemkin gave the agent access. The critics are right that you can call all of this user error, and technically they are right. But that framing obscures the actual question. If agents require perfectly scoped permissions, perfectly reviewed plans, and perfectly written constraints to avoid disasters, they are not reducing operational risk. They are adding a new surface for operational risk, one that is harder to reason about because the agent’s decision-making is not transparent until after something goes wrong.

The reasonable counterargument is that the tooling will mature: sandboxing, least-privilege enforcement, explicit approval gates for destructive operations, plan-only modes that require human sign-off before any write. These are real mitigations and they already exist in some form. Replit added a planning-only mode after the Lemkin incident. Terraform has destroy protection. AWS has guardrails. The argument is not that agents can never be used in production environments with careful controls. The argument is narrower: an agent with autonomous write access to production, without an enforced gate on destructive operations, is a liability. Not because the agent is trying to cause harm, but because it is trying to complete the task, and in production, those are sometimes the same thing.

You do not let a junior engineer deploy to production on day one. That restriction exists not because junior engineers are stupid, but because production is a place where the cost of a mistake is not bounded by the mistake itself. The same logic applies to any new operator, human or otherwise, that has not demonstrated it can be trusted with the specific power it is being given. The current default in too many teams is to give agents what they need to get the job done and add guardrails later. The incidents above are what later looks like.

Rohana Rezel is a technologist, researcher, and community leader based in Vancouver, BC.

Discuss this article on boreal.social