Agentic Fabriq
AGENT SECURITY
The Real Cost of Rogue AI Actions: What Companies Are Seeing
The best-documented version of this failure cost $460 million in 45 minutes and had nothing to do with AI. What that case teaches about blast radius, which actions actually deserve a gate, and where the bill lands.
TL;DR
Rogue AI actions are rarely dramatic, and that's what makes them expensive. The cost isn't one catastrophic event; it's an ordinary agent doing ordinary work with slightly too much reach, discovered after the fact.
The best-documented version of this failure has no AI in it at all. On 1 August 2012 Knight Capital deployed code that left a dead subsystem callable, and in about 45 minutes an automated router sent 4 million executions for more than 397 million shares and lost more than $460 million. Warning messages fired 97 times before the market opened. Nobody had wired them to anything.
We think the industry keeps reaching for the wrong fix. Upgrading the model doesn't change what happens when an agent is trusted with an irreversible action and no boundary around it. A smarter model executes the same mistake faster and with more confidence.
This isn't an argument for gating everything. A gate on every action trains people to rubber-stamp, and the regulation that mandates human oversight asks specifically for the ability to "intervene" and to halt "in a safe state," not for a confirmation dialog on every read.
The bill also isn't one line. It shows up as the hours spent undoing the action, the review step every team bolts on afterward, and the pilot that quietly gets shelved. Scope the reach and you remove all three.
Overview
Most coverage of AI incidents reaches for the biggest available story. That isn't where the cost accumulates inside most organisations. It comes from something less cinematic: an agent that did exactly what it was told, a little too literally, to a scope a little too wide.
Take a knowledge-assistant agent asked to clean up a shared workspace by archiving anything untouched in a week. Nothing about the instruction is unusual and nothing about the agent is malfunctioning. It has no concept that "clean up" and "delete permanently" are different operations, and the integration it was wired into never distinguished them either. The cost isn't a breach. It's a Tuesday afternoon that turns into a week of reconstruction.
OWASP named this failure class. Excessive Agency, LLM06 in the 2025 Top 10 for LLM Applications, is "the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, regardless of what is causing the LLM to malfunction," split into excessive functionality, excessive permissions and excessive autonomy. Read the last clause carefully. The cause of the malfunction is out of scope, so the defence cannot depend on diagnosing it.
The question that surfaces after the fact is always the one that should have been asked before it. "Who gave the agent permission to do that?" arrives once the damage is done, which means the boundary that mattered was never about catching the mistake. It was about not granting the reach.
Why the Headline Isn't the Real Cost
A single catastrophic incident is rare enough that most teams reasonably discount it. The smaller version is not rare: a bulk action against the wrong scope, a filter that silently falls through to "select everything" when a query returns nothing, an agent that reads "standardise the categories" as licence to rewrite every automation attached to them. None of that requires an attacker. It requires more reach than the situation warranted and no check between "the agent decided to act" and "the action executed." We think that gap, not model capability, is the subject of every incident in this space.
The cost data that exists measures the breach-shaped subset. IBM's Cost of a Data Breach report puts the global average at $4.99M, "a 12% increase over last year and a record high," and records a "56% increase in AI-driven attacks." It cannot price the category here, because an agent deleting the wrong records is not a breach and never gets counted.
Two Ordinary Failures
Take a support-ticketing bot asked to standardise category labels across a queue. "Standardise" sounds precise and specifies almost nothing operationally, so the agent improvises: it reassigns priorities, edits routing rules, and closes tickets it decides no longer match a category. Nobody told it to touch routing. Routing was reachable, and reachable things get touched when an instruction is ambiguous and nothing stops the interpretation from running.
Take a second hypothetical: a marketing agent connected to the CRM to nudge recent signups. A segment filter returns zero rows, falls through to "no filter applied," and the follow-up goes to the whole database, churned accounts, internal staff and a competitor included. No vulnerability to patch, just a compliance flag to check and trust to rebuild in a system that did nothing outside its permissions.
Both are invented, not incidents anyone reported to us. The pattern is identical: a vague instruction, a boundary wide enough to let the vague reading through, and no step that asked whether the action should run.
The Best-Documented Case Has No AI In It
The best-documented example of this failure mode involves no model at all. The SEC's 2013 order against Knight Capital Americas reads like a design review of everything that bounds an autonomous system, written from the wreckage.
Knight deployed new code to its order router in stages from 27 July 2012, replacing an unused subsystem called Power Peg that had been discontinued in 2003, and repurposing the flag that used to activate it. Power Peg "remained present and callable." One of eight servers did not get the new code. On the morning of 1 August, orders carrying the repurposed flag reached that server and woke the dead subsystem, which had been quietly broken since 2005 when its cumulative-quantity counter was moved earlier in the sequence and never retested.
The numbers are in the order. For 212 incoming parent orders, the router sent "millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes," accumulating a roughly $3.5 billion net long position and a $3.15 billion net short one. Knight "realized a $460 million loss" and paid a $12,000,000 civil money penalty.
The detail we would put on a slide is not the loss. It is that the system was shouting. An internal process generated automated emails naming the error "Power Peg disabled" and sent 97 of them to Knight staff before the 9:30 open. The SEC's finding: "Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them."
Figure 1 — Knight Capital, 1 August 2012. Read it as a checklist of the controls an autonomous system needs, each one absent.
Swap the order router for a tool-calling agent and the chain is unchanged. Reachable capability nobody meant to leave reachable. A signal that went nowhere. No bound on how much one run could do before a human could intervene. The SEC's summary finding is the sentence to steal: Knight "did not have a system of risk management controls and supervisory procedures reasonably designed to manage the financial, regulatory, and other risks of market access."
Replit's own writing about agents and live data points at the same structural fix. Its "Safe vibe coding" post names the worry, that the "scariest failure mode we see in vibe coding" is "when AI accidentally erases git history, backup data, or other key files that can't be recovered once gone," and the remedy it shipped was environment separation: "the ability for each app to have two databases: one for development, and one for production." Not a better model. A smaller blast radius.
Why Model Quality Isn't the Variable
What's absent from every failure above is a bad model. A more capable model executes the same flawed instruction faster and with more confidence, not more caution. Capability isn't the safety lever. Scope is.
Microsoft's AI Red Team made the same split in its April 2025 taxonomy of agentic failure modes, separating "novel" failures unique to agentic systems from "existing" ones. Its memory-poisoning entry shows why model quality is the wrong dial: the problem is that "the absence of robust semantic analysis and contextual validation mechanisms allows malicious instructions to be stored, recalled, and executed," and the mitigation is structural, "requiring external authentication or validation for all memory updates."
The agents in these scenarios weren't confused about how to wipe a workspace or send a few thousand emails. They were good at it. What was missing was a step that asked whether the action should happen, and a blast radius small enough that a wrong answer wouldn't be expensive to undo.
Power without a boundary is expensive, and the expense doesn't scale with how smart the system is. It scales with what the system was allowed to touch.
The Three-Layer Bill
One permitted action generates cost in three places, and only the first shows up on a timesheet.
Figure 2 — One permitted action, three cost layers. Only the first is billed cleanly.
The direct layer is the hours spent reconstructing what happened and undoing it, which is why it gets mistaken for the whole bill.
The operational layer comes after: a burned team adds a review step to everything the agent touches, erasing much of the efficiency the automation was meant to deliver. The strategic layer is the most expensive and hardest to see coming. Executives who watched one rogue action unfold turn risk-averse about the next ten reasonable ones, and a promising pilot gets shelved not because it failed technically but because nobody wants to sign off on the next mishap.
There is a fourth layer inside the EU AI Act's high-risk scope, and it is the one with published numbers. Article 99 sets fines of up to €35,000,000 or 7% of total worldwide annual turnover for prohibited practices under Article 5, up to €15,000,000 or 3% for non-compliance with other operator obligations, and up to €7,500,000 or 1% for supplying misleading information, each "whichever is higher."
The same regulation prices the evidence problem. Article 12(1) requires high-risk systems to "technically
allow for the automatic recording of events (logs) over the lifetime of the system," and Article 19
requires providers to keep them "for a period appropriate to the intended purpose" and "of at least six
months," from 2 December 2027 for Annex III systems. An organisation that cannot reconstruct which agent
did what, for whom, pays the direct layer twice: to fix the damage, then to prove what happened. That
record is aact:audit-granularity/per-action-with-principal-chain in the
Agent Access Control Taxonomy.
Which Actions Deserve a Gate
"Add human review" is where most teams land after an incident, and it is how the operational layer gets
expensive. The narrower question is which actions. Two properties decide it: reversibility, whether an undo
restores the prior state without reconstruction, and fan-out, how many records or recipients one call
reaches. A gate earns its place where both are bad: the taxonomy's
aact:human-oversight/approval-for-high-risk, not its approval-for-all.
| Action shape | Reversible? | Fan-out | Treatment |
|---|---|---|---|
| Read a record | Yes | 1 | No gate. Log it |
| Update in place, with history | Yes | Low | No gate. Version history is the undo |
| Archive or soft-delete | Yes | High | No gate, but cap the batch and log the set |
| Hard delete, or delete with no retention window | No | Any | Gate, and enumerate the target set first |
| External email, or a post to a public channel | No | High | Gate on first use per audience, then allow-list it |
| Bulk write across a table or workspace | Only at reconstruction cost | Very high | Gate, after a dry run returning the affected count |
| Schema, permission or credential change | No | Structural | Keep it out of the agent's tool list where possible |
Table 1 — The two properties that decide whether a gate is worth its cost.
The contested row is the third. Archiving is reversible, so it does not need a person, but it needs a ceiling. The workspace-cleanup scenario above is what happens when a reversible operation with no cap meets an integration where archive and delete were never distinguished.
The EU AI Act's human-oversight article is specific about what oversight means, and it is not a confirmation dialog. Article 14 requires high-risk systems to be designed so they "can be effectively overseen by natural persons," and lists what the overseer needs: to "disregard, override or reverse the output," and to "intervene in the operation" including through "a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state." It also names what a rubber-stamp gate produces, requiring overseers to stay aware of "the possible tendency of automatically relying or over-relying" on outputs.
A gate on every action is a gate on nothing. The regulation asks for the ability to stop and reverse, not for a confirmation prompt, and Knight's 97 unread emails are what a signal nobody can act on is worth.
Telling the Failures Apart
After the fact these failures look the same: an agent did something expensive and every call was authorised. They have different fixes, so it pays to tell them apart from the record you already have.
| What happened | The tell | The fix that applies |
|---|---|---|
| Ambiguous instruction, agent guessed wide | The action fits a plausible reading of the prompt, and the affected set is defined by what was reachable rather than what was asked | Narrow the tool list; name the target set before a destructive call |
| Filter fell through to "everything" | The affected set equals the whole table or workspace, exactly | A dry run returning the affected count, and a cap that rejects it |
| Retrieved content carried an instruction | The instruction sits in data the agent read, not the user's message | Provenance on retrieved content; no destructive tools in that loop |
| Agent acted with reach the invoking user lacks | The action exceeds the acting user's own entitlement | Effective permission as an intersection, not a maximum |
| Stored memory told the agent to do it | The behaviour persists across sessions with no matching prompt | Microsoft's recommendation: external validation for memory writes |
| Credential reached somewhere it shouldn't | The secret appears in a log, a prompt or a tool argument | Server-side injection, so the agent never holds it |
Table 2 — Six causes that all present as an authorised, expensive action.
Rows one and two get conflated constantly and have opposite fixes. A wide guess is a tool-surface problem: the agent should never have reached routing rules. A fall-through is arithmetic: the affected count was knowable before the call and nobody looked. Human review fixes neither, because a reviewer shown "update 40,000 records" approves it about as often as one shown "update 12."
What Actually Bounds the Risk
None of the fixes that work require a better model. OWASP's mitigation list for Excessive Agency is a fair summary of the field: minimise extensions and their functions, avoid open-ended tools, restrict permissions, execute in the user's context, require human approval for high-impact actions, implement complete mediation, sanitise inputs and outputs, and monitor with rate limiting.
Four earn the ranking. Keep production separated from experimentation until an agent has been tested against a copy of what it will touch, the fix Replit shipped. Execute in the user's context, so the agent's reach cannot exceed the reach of whoever invoked it. Gate the bulk, irreversible, high-fan-out actions from Table 1 and leave the rest fast. And rate-limit, because fan-out over time is the same problem as fan-out per call, which is what Knight's 45 minutes illustrates.
Then make the signals actionable. The Knight order is a study in instrumentation wired to nothing, and the agent equivalent is an audit record nobody reads. OWASP's Agent Control Standard, published September 2026, puts the requirement as agents being "inspectable, traceable and instrumentable," because "enterprises cannot rely on black-box agents operating across cloud, SaaS, on-premises and endpoint environments."
We'd add one pre-launch question: handed to a new employee on their first day, would you let them execute this instruction unsupervised? A "no" paired with an agent that already has the access is the gap behind every story here.
Where Fabriq Fits
Agentic Fabriq addresses the reach half of this, not the judgment half. It holds credentials per user in a vault and attaches them server-side at the moment of the call, so the agent never handles the secret, and an agent's effective tool list is the intersection of the user's scopes and the agent's rather than the union. Every request carries two identities, the agent and the acting user, into an audit record, which is what makes row four of Table 2 answerable.
What it does not do is decide which actions are too dangerous to run unattended. There is no approval step and no provider-side revocation; clearing a vault copy of a credential is not the same act as revoking the grant upstream, and that call is still yours. The gates in Table 1 are a design decision about your own tools.
Frequently Asked Questions
What is a rogue AI action? In practice, an authorised action taken by a working system with more reach than the situation needed. OWASP's Excessive Agency category is the closest formal definition: damaging actions from "unexpected, ambiguous or manipulated outputs," independent of what made the model go wrong.
How much does an AI incident cost? The breach-shaped subset has numbers: IBM's report puts the global average data breach at $4.99M. An agent deleting or sending the wrong thing has no published average, because it is not a breach and rarely leaves the organisation. Knight's $460 million in 45 minutes is the best-documented ceiling for an unbounded automated system.
Which agent actions should require human approval? The irreversible, high-fan-out ones in Table 1: hard deletes, external sends, bulk writes, and structural changes to schemas, permissions or credentials. Gating reads and versioned edits buys nothing and trains reviewers to click through.
Does the EU AI Act require a human in the loop? For high-risk systems it requires effective human oversight under Article 14, including the ability to "disregard, override or reverse the output" and to halt the system "in a safe state." That is a design obligation, not a mandate to confirm every action.
How long do we have to keep agent logs? Under Article 19, providers of high-risk systems keep automatically generated logs for "at least six months," from 2 December 2027 for Annex III systems. Sector rules run longer. We would exceed the minimum anyway, because the question you get asked is about an incident six months old.
We already log everything. Isn't that enough? Knight logged everything. It sent 97 messages naming the exact fault before the open, and the SEC found they were never designed to be alerts and "Knight personnel generally did not review them." A log wired to nothing is a record of the incident, not a control on it.
Conclusion
Most rogue AI actions never make the news. What they make is a dent in trust, rollout velocity, and the confidence of whoever has to explain the next automation proposal. The financial picture is a habit rather than an event: undoing things, one review step at a time, until the automation costs more attention than it saves.
The one case with a full public record is not about AI at all, and that is the point. Knight lost $460 million in 45 minutes because a capability nobody meant to leave reachable was reachable, a warning nobody had wired to an alert fired 97 times, and nothing bounded how much a single run could do. Each of those is a decision made before the incident.
None of this is a reason to stop building agents. It's a reason to decide, before the agent runs, how much it's allowed to break.
The companies that get this right won't have smarter models than everyone else. They'll have agents that can't reach further than the job actually requires.
Sources
All URLs read 2026-09-29.
- In the Matter of Knight Capital Americas LLC, SEC Release No. 34-70694 — the 1 August 2012 events: the Power Peg code left present and callable, 212 parent orders producing 4 million executions in 154 stocks for more than 397 million shares in roughly 45 minutes, the $460 million loss, the 97 unread "Power Peg disabled" messages, and the $12,000,000 penalty.
- LLM06:2025 Excessive Agency, OWASP Top 10 for LLM Applications — the definition, the split into excessive functionality, permissions and autonomy, and the mitigation list quoted in the guardrails section.
- Article 14: Human Oversight, EU AI Act — the design obligation for effective oversight, the overseer's ability to disregard, override or reverse an output, the stop button and safe-state halt, and the automation-bias requirement.
- Article 99: Penalties, EU AI Act — the €35,000,000 or 7% ceiling for prohibited practices, €15,000,000 or 3% for other operator obligations, and €7,500,000 or 1% for misleading information.
- Article 12: Record-Keeping, EU AI Act — the requirement that high-risk systems technically allow automatic recording of events over the system's lifetime.
- Article 19: Automatically Generated Logs, EU AI Act — the at-least-six-months retention floor and the 2 December 2027 and 2 August 2028 commencement dates.
- Cost of a Data Breach Report, IBM — the $4.99M global average and its 12% year-on-year record high, the 56% increase in AI-driven attacks, and the AI model inversion figure.
- Safe vibe coding, Replit — the "scariest failure mode" statement about irrecoverable data, and the shipped separation of development and production databases.
- New whitepaper outlines the taxonomy of failure modes in AI agents, Microsoft Security Blog — April 2025; the novel-versus-existing and safety-versus-security split, and the memory-poisoning description and mitigation.
- Agent Control Standard, OWASP GenAI Security Project — September 2026; agents as inspectable, traceable and instrumentable, and the argument against black-box agents across cloud, SaaS and endpoint environments.
- OWASP Top 10 for Agentic Applications 2026 — published 9 December 2025; the current risk taxonomy for agentic applications.
- Agent Access Control Taxonomy — the audit granularity and human oversight dimensions, and the questions they ask: what is recorded and who the action is attributed to, and when a person gets involved.
- Agent access control taxonomy, human-oversight.yaml —
aact:human-oversight/approval-for-high-risk, where only actions matching a risk rule wait for a named human, againstapproval-for-allandpost-hoc-review.