Building Trust in Autonomous Systems

Overview

The instinct when introducing an autonomous system is to say it's safe. That is the wrong starting point. What works is an operating model where trustworthiness is something a skeptical outsider can check.

NIST's AI RMF names seven characteristics of a trustworthy AI system: "valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed". It also says something that gets lost when the list is quoted: these characteristics trade against each other. "Highly secure but unfair systems, accurate but opaque and uninterpretable systems, and inaccurate but secure, privacy-enhanced, and transparent systems are all undesirable. A comprehensive approach to risk management calls for balancing tradeoffs among the trustworthiness characteristics."

What follows covers the operational subset of that list, the part a platform team owns rather than a model team: bounded behavior, transparency, consistency, recoverability, and lifecycle discipline. Fairness and explainability matter enormously and belong to a different conversation about the model itself.

A claim about an agent is worth exactly as much as the record behind it, and for most agents running today, that record doesn't exist yet.

Trust Is Not a Claim

You can't declare trust into existence by saying so. It shows up in what a team can produce when someone asks a hard question about the agent.

Consider two procurement agents that both negotiate and place supplier orders. The first is described by its team as "carefully tested and reliable." The second has a documented purpose, a spend ceiling enforced in code, scoped credentials that only reach the procurement system, an audit trail of every order, and a named owner in procurement. The first is a claim. The second is an operating model. When a duplicate order goes out, only one of them can be investigated.

Figure 1 — Same incident, two agents. The claim has nothing to check itself against; the operating model does.

NIST's framing is sharper than ours. "A transparent system is not necessarily an accurate, privacy-enhanced, secure, or fair system. However, it is difficult to determine whether an opaque system possesses such characteristics, and to do so over time as complex systems evolve." Transparency does not make an agent good. It makes the question answerable.

Bounded Behavior

An autonomous system is trustworthy when its behavior is bounded. Teams routinely confuse a wider blast radius with more autonomy, when what they've built is something less predictable. The tighter and more explicit the edges, the more autonomy you can responsibly hand over.

The specs that actually govern this are not AI specs. NIST SP 800-207's third zero-trust tenet says access is granted per session and "with the least privileges needed to complete the task", and then adds the sentence that matters most for a tool-calling agent: "authentication and authorization to one resource will not automatically grant access to a different resource." An agent holding one broad token violates that on every call. RFC 9700, the OAuth 2.0 Security Best Current Practice, says the same thing about the token itself in section 2.3: "The privileges associated with an access token SHOULD be restricted to the minimum required for the particular application or use case."

The July 2026 MCP specification turns it into wire behaviour. An MCP server, acting as an OAuth 2.1 resource server, "MUST validate that access tokens were issued specifically for them as the intended audience", and the specification closes the obvious workaround in four words: "MCP servers MUST NOT accept or transit any other tokens." It also defines what a boundary should look like when it bites. A call with a token that lacks the needed scope gets an HTTP 403 with error="insufficient_scope" and a scope parameter naming what was missing.

HTTP/1.1 403 Forbidden
WWW-Authenticate: Bearer error="insufficient_scope",
                         scope="files:write",
                         resource_metadata="https://mcp.example.com/.well-known/oauth-protected-resource",
                         error_description="File write permission required for this operation"

That response is the shape of a trustworthy boundary, aact:failure-behavior/refuse-and-explain in the Agent Access Control Taxonomy: it refuses, it says what was refused, and it says what would have been sufficient. A boundary that silently degrades or returns an empty result teaches the agent nothing and leaves no record of the attempt.

Enforcement point Enforcement point computes user scopes INTERSECT agent scopes in scope: fetch audience-restricted token from vault, call the SaaS API, return the result, secret never reaches the agent out of scope: return 403 insufficient_scope with the missing scope named, and RECORD THE DENIAL The denial branch is the one most implementations leave unlogged. -->

Figure 2 — Both branches matter. The refusal path is what turns a documented scope into evidence that the boundary is doing work.

Take a data-analytics agent that generates and runs SQL against the warehouse. Bounded means read-only credentials, an audience covering only the curated reporting schemas, and a 403 on anything else.

Transparency

People calibrate their reliance on a system correctly only when they can see what it's allowed to do. A marketing agent that drafts and schedules campaign emails should make its own hand visible in the send record, not present the message as though a person typed it.

The EU AI Act treats the failure to do this as a design defect rather than a courtesy. Article 14(4) requires that whoever oversees a high-risk system be able to "remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)". You cannot stay aware of a tendency you have no signal about. An agent whose output is indistinguishable from a colleague's has removed the signal.

NIST separates three things people collapse into one word. "Transparency can answer the question of 'what happened' in the system. Explainability can answer the question of 'how' a decision was made in the system. Interpretability can answer the question of 'why' a decision was made by the system." Operational trust needs the first one and can survive without the other two. Most teams reach for the third, which is the hardest, and skip the first, which is a logging decision.

Consistency

An agent that behaves correctly nine times and unpredictably the tenth isn't trusted, and shouldn't be. This is the property with the best public measurement, and the measurement is unflattering.

τ-bench introduced pass^k, which scores an agent as successful only if it succeeds on all k independent attempts at the same task. The authors report that "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)". Read that as two numbers rather than one. The first is capability. The gap between them is variance, and variance is what a demo hides.

Table 1 — Four reliability measurements. Only the third and fourth are worth putting in front of a security review.

Consistency comes from permissions enforced at the boundary, not from hoping the model stays in character. An HR onboarding agent that provisions accounts should have a scope stating which systems it can grant access to and at what level. If it attempts a role outside that scope, the attempt should be blocked and flagged rather than logged as a quiet success.

We think this is the part most teams get backwards. They write the behavior into the prompt and treat the scope as documentation, when it should be the other way around. The point isn't that agents never deviate. It's that deviation is caught, recorded, and reviewable.

Recoverability

Trustworthy systems aren't the ones that never fail. They're the ones you can recover from when they do.

• Investigate: reconstruct exactly what the agent did, with what permission, on whose behalf.
• Disable: cut the agent's access immediately, without waiting for a redeploy.
• Update: tighten the scope so the same failure can't recur.
• Roll back: reverse the affected actions where the action is reversible.
• Improve: feed the incident back into the agent's design and guardrails.

NIST SP 800-61r3 names the thing that makes the second bullet real. An incident response policy should record "roles, responsibilities, and authorities, such as which roles have the authority to confiscate, disconnect, or shut down technology assets". Authority to disconnect, held by somebody specific, before the incident.

Imagine an IT-operations agent that misreads an alert and scales down a production cluster. Recoverability means an engineer can pull the action log, see the decision and the credential it used, disable that credential on the spot, restore capacity, then narrow the scope so one noisy alert can't trigger a destructive change again. Without those levers, the same incident is a guessing game.

Lifecycle Discipline

Agents shouldn't live forever by default. Business needs change, integrations are deprecated, and the assumptions an agent was built on expire. An agent still running long after its purpose faded isn't an asset. It's an unowned access path nobody is watching.

NIST AI RMF makes this its own governance requirement rather than an afterthought. GOVERN 1.7 asks for processes "for decommissioning and phasing out AI systems safely and in a manner that does not increase risks or decrease the organization's trustworthiness". A badly handled retirement can lower trustworthiness, so the exit needs as much design as the launch.

Picture a legal-review agent stood up to triage contracts during a one-time migration. The disciplined outcome when the migration ends is to retire it: revoke its credentials, remove it from the registry, close out its ownership. The undisciplined outcome is that it sits idle for a year with live access to the contract repository, exactly the kind of artifact an attacker or an auditor finds first.

The Model-Quality Trap

Reaching for a better model is the easy move. It's a line item someone can approve without an argument, and it looks like progress on a status update in a way that "we scoped the credentials" never quite does.

The prompt-injection literature is where this gets settled, because it is the one place the two approaches have been measured against each other on the same benchmark. AgentDojo puts agents in 97 realistic tasks with 629 security test cases and reports that state-of-the-art models "fail at many tasks (even in the absence of attacks)". CaMeL, which is not a model improvement at all but a system design splitting a privileged planner from a quarantined data-handling model and tracking provenance through a custom interpreter, reports solving "77% of tasks with provable security (compared to 84% with an undefended system)" on that benchmark.

The defence costs seven percentage points of capability and buys a security property the undefended system cannot have at any model size, because the property is structural. No upgrade produces it. A sharper model changes how well the agent reasons about the task in front of it. It says nothing about who's allowed to see what the agent did afterward, or whether anyone would notice it reaching further than it should.

How Trust Claims Fail

The interesting failures are the ones where the governance story still reads well.

The control plane fails open. The OWASP Agent Control Standard defines a Guardian that inspects an agent's intended action and returns allow, deny, or modify. Its reference implementation defaults to proceed, which is aact:failure-behavior/fail-open, and its README is blunt about the consequence: "A Guardian that crashes, hangs, or is unreachable stops governing, silently, and the host proceeds as if nothing asked." Symptom: nothing. Test by killing the Guardian and confirming the next call is refused.

The audit record has one identity, not two. Symptom: the log answers "which agent" and not "on whose behalf". Every question a reviewer actually asks is about the second one, and it cannot be reconstructed after the fact.

The deny rate is zero. Symptom: a scope that has never refused anything. Either the agent never exceeded it, which is worth knowing, or refusals are not recorded, which means the scope is documentation. A boundary you cannot see working is indistinguishable from a boundary that isn't.

The token has the wrong audience. Symptom: a call succeeds against a system the agent was never provisioned for, because a broadly scoped token was passed along. This is precisely the case MCP forbids with "MUST NOT accept or transit any other tokens", and the reason audience validation is a server-side duty rather than a client-side courtesy.

Reliability was measured once. Symptom: a green demo and a pass^8 nobody ran. Two agents with the same pass@1 can differ sharply on pass^8, and only one is safe to leave unattended.

The first and third look identical from the outside. An agent that acted because the Guardian was down, and one that acted because its scope never checked, both present as "no denial in the log". Telling them apart needs a health signal for the enforcement point itself, which is the piece most architectures draw and few instrument.

In Practice

Trust gets built incrementally, starting with the controls that turn an opaque agent into one whose behavior can be inspected and bounded. A useful way to gauge where an agent stands is to ask the questions an investigator would ask, before any incident forces them.

• Identity and ownership: does this agent have its own identity, and is there a named owner accountable for it?
• Access and permissions: can you state exactly what it can reach, and is that the minimum the job requires?
• Evidence: if asked what the agent did last Tuesday, could you produce a record: action, tool, data, acting user, and whether the check passed?
• Containment: if it misbehaved right now, how fast could you disable its access and reverse what it touched?
• End of life: is there a point at which this agent gets reviewed or retired, or does it simply persist?

The third question has a statutory floor attached where the agent falls under the EU AI Act. Article 12 requires high-risk systems to "technically allow for the automatic recording of events (logs) over the lifetime of the system", and Article 26(6) requires deployers to keep those logs "for a period appropriate to the intended purpose ... of at least six months". Following the Digital Omnibus, those obligations apply from 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I ones, per the European Commission's regulatory-framework page. A lot of writing still says August 2026. It is wrong.

Where Fabriq Fits

Agentic Fabriq holds credentials per user in a vault and attaches them server-side at the moment of the call, so the agent never handles the secret. Every call carries two identities, the agent and the acting user, which is what makes the second bullet in the checklist above answerable rather than aspirational, and an agent's effective tool list is the intersection of the user's scopes and the agent's, so widening one does not widen access on its own. Refresh runs automatically behind that boundary with last-used tracking.

What it does not do is reach into the provider for you. Clearing a vault copy is not the same act as invalidating a grant upstream, so revoking at the provider remains yours to do.

Frequently Asked Questions

How do you measure trust in an autonomous system? You don't measure trust. You measure the things that make it checkable: whether the scope refuses what it should, whether the log answers "on whose behalf", whether an owner can disconnect the agent, and whether the agent passes pass^k rather than pass@1.

What makes an AI agent trustworthy? Bounded behavior, transparency about what it may do, consistent enforcement at the boundary, a recovery path, and a defined end of life. NIST's fuller list adds validity, safety, security, privacy and fairness, and warns that these characteristics trade against one another rather than stacking.

Is a more capable model a more trustworthy one? Not by itself. CaMeL's 77% with provable security against 84% undefended is the cleanest illustration: the security property came from architecture, not scale, and cost capability to get.

Does human oversight make an agent trustworthy? Only if the overseer can act. The AI Act pairs oversight with the ability to "disregard, override or reverse the output" and to "interrupt the system through a 'stop' button". Oversight without either capability is a signature.

What should an agent audit log contain? Action, tool, resource, agent identity, acting user, the authorization decision including denials, and a timestamp. The denial is the field most often missing and the one that proves the boundary exists.

How long should we keep agent logs? At least six months where the AI Act applies, under Article 26(6), and longer wherever sector rules say so. Retention shorter than a quarter means most investigations will start after the evidence has gone.

Can we trust an agent we cannot explain? Often, yes, and this is where the vocabulary matters. NIST separates transparency ("what happened") from explainability ("how") and interpretability ("why"). Operational trust runs on the first. An unexplainable agent with a complete action record is in much better shape than an explainable one with no record.

What is the cheapest thing to do first? Make the scope an enforcement point and log the denials. It is one change, it produces the evidence every other property depends on, and it does not require a model decision.

Conclusion

Trustworthy autonomy is the product of deliberate engineering and governance, not optimism. It comes from behavior that's bounded, interactions that are transparent, conduct that's consistent, failures that are recoverable, and a lifecycle that's disciplined. Each of those is something you build and operate.

Model quality will keep improving, and that's welcome. It will never substitute for knowing which agents exist, what they can do, who owns them, and what they actually did. We'd rather ship a narrower agent with real boundaries than a capable one with none, and the pass^8 numbers are the reason: an agent that succeeds most of the time is an agent that needs a boundary, not a better prompt.

Sources

All URLs read 2026-09-29.

• Agent Access Control Taxonomy — the failure-behavior dimension, which asks what happens when an authorization decision cannot be made: fail open, fail closed, or refuse and explain.
• NIST AI 100-1, AI Risk Management Framework 1.0 — the seven trustworthiness characteristics, the tradeoff warning, "accountability presupposes transparency", the what/how/why split, and GOVERN 1.7 on decommissioning.
• NIST SP 800-207, Zero Trust Architecture — tenet 3 on per-session access and least privileges, and that authorization to one resource does not grant access to another.
• NIST SP 800-61r3, Incident Response Recommendations — April 2025; the requirement to record which roles hold authority to disconnect or shut down assets.
• RFC 9700, Best Current Practice for OAuth 2.0 Security — section 2.3 on restricting access token privileges to the minimum required, and audience restriction.
• MCP specification 2026-07-28, Authorization — audience validation as a server duty, the prohibition on accepting or transiting other tokens, and the 403 insufficient_scope challenge format.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — the pass^k metric and the under-50% pass@1 / under-25% pass^8 retail result.
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses — 97 tasks and 629 security test cases, and that current models fail many tasks with no attack present.
• Defeating Prompt Injections by Design (CaMeL) — 77% of AgentDojo tasks solved with provable security against 84% undefended.
• EU AI Act Article 12, Record-keeping — automatic logging over the lifetime of a high-risk system.
• EU AI Act Article 26, Obligations of deployers — the at-least-six-months log retention floor.
• EU AI Act Article 14, Human oversight — automation bias, the override and reverse capability, and the stop button.
• European Commission, AI Act regulatory framework — the post-Digital-Omnibus dates: 2 December 2027 for Annex III, 2 August 2028 for Annex I.
• OWASP Agent Control Standard — the Guardian allow/deny/modify contract and its published fail-open default posture.