AI Governance Has to Move From Policy to Runtime

Overview

Most enterprise AI policy was written for systems that only talked. It tells employees not to paste sensitive data into unapproved tools, not to trust outputs without review, not to deploy anything without a security sign-off. None of that is wrong, and none of it says what happens the moment an agent can act on its own.

An agent connected to Gmail, Salesforce, and a knowledge base doesn't just answer questions. It reads records, drafts messages, and in a growing number of deployments, sends them. The policy document has an opinion about whether that's allowed. The tool-calling path, in most organizations, has none.

The layer that has to sit between the two is built from agent identity, permission boundaries, approval gates, context-aware decisions, and an audit trail that can reconstruct what actually happened. None of these are abstract. Each maps to a specific point between a request and an action, and each is missing from most agent deployments today.

Motivation

The first wave of enterprise AI was copilots and chat interfaces. They were useful, and the human stayed the execution layer: a model proposed, a person decided. Agentic systems move the agent into that execution layer. It can hold credentials, call APIs, and chain steps across systems that used to be reserved for a person, or for code an engineer reviewed before it shipped.

That creates a real control problem, not a hypothetical one. Traditional software fixes the set of reachable states at design time: engineers decide which buttons exist and which roles can press them, and someone reviews that before release. An agent can compose a sequence of actions nobody explicitly wrote down, generated fresh from the user's request, the tools available, and whatever context it retrieved. The unit of governance has to shift from the screen a person can reach to the specific action an agent is attempting, in the moment it attempts it.

Take a support agent connected to Zendesk, Slack, and a knowledge base. Some of what it does is low risk: summarizing a ticket, drafting a reply. Some of it isn't: escalating an issue, notifying a channel the whole company reads, or including a customer's account details in an outbound message. A policy that says "agents should be used responsibly" doesn't tell the runtime which of those needs a human and which doesn't.

The Runtime Governance Layer

Runtime governance is the layer that decides what an agent is actually allowed to do, evaluated at the moment it tries to do it. It has to weigh several things at once: who invoked the agent, which agent is acting, which tool and data are involved, what the action would do, and the rules attached to that particular workflow.

A plain allow-or-deny model runs out of expressiveness fast. Real deployments need finer outcomes: allow a read but not a write, allow a draft but block automatic send, redact a field, or let an action through while flagging it for later review. Mature access-control systems already separate the point where a decision gets made from the point where it gets carried out. A decision point evaluates the request against the relevant rules; a separate enforcement point sits in the agent's tool-calling path and acts on that verdict before the side effect occurs. Keep those two apart and the rules can change without rewiring every agent.

We think the clearest way to see the shape of it is to picture every tool call passing through a check before it reaches the underlying system.

Figure 1 — A tool call passing through a runtime decision before it reaches Gmail, the CRM, or the database. The exact rules matter less than the fact that every branch gets evaluated and logged before the side effect happens.

# Pseudocode for a runtime decision on a tool call
def evaluate(request):
    # request carries the full context of the attempted action
    #   request.user, request.agent, request.tool
    #   request.action      -> read | write | send | delete
    #   request.data_class  -> public | internal | confidential | pii
    #   request.destination -> internal | external recipient

    if not user_may_invoke(request.user, request.agent):
        return Decision.DENY

    if request.action == "send" and request.destination == "external":
        if request.data_class in ("confidential", "pii"):
            return Decision.DENY
        return Decision.REQUIRE_APPROVAL

    if request.action in ("write", "delete"):
        return Decision.REQUIRE_APPROVAL

    if request.data_class == "pii":
        return Decision.ALLOW_WITH_REDACTION

    return Decision.ALLOW  # always recorded in the audit log

The exact rules will always differ by organization, and almost no real deployment expresses them as one function this clean. What has to hold is the shape: every action gets evaluated in context, and the result can be deny, allow, require approval, redact, or allow-and-log.

Core Capabilities

A layer like this is built from a small number of capabilities that work together. Each answers a different question: who's acting, what may they do, when does a human need to step in, does context change the answer, and can the decision be reconstructed later.

Agent Identity

The foundation is knowing which agent is acting, not just which user or application. Without a distinct identity, agent actions blur into ambiguity: a log shows a user performed an action when the user really delegated it, or shows an application called an API with no record of which agent initiated the request.

Most of this gets worse because of how agents authenticate today. A shared service account or a broad API token collapses every agent and every user behind one identity at the exact moment that matters. If an agent acting for a junior employee and one acting for an admin present the same credential, the downstream system can't tell them apart, and neither can the audit trail.

Agent identity means treating agents as first-class actors: created, reviewed, monitored, and revoked through an actual governance process, with the chain of delegation preserved from the human who initiated a request through the agent that acted to the system that was touched.

Permission Boundaries

An agent shouldn't automatically inherit everything the user or application it operates through can do. It should get the minimum set of capabilities its workflow actually needs, and that requires thinking more granularly than "it can connect to Salesforce." Can it read records, or update them? Retrieve documents, or share them externally? Query a database, or modify tables?

The risk of an agent comes far more from the specific action than from the tool name. A read-only connection to a customer database and one that can run arbitrary updates might both show up as "database access" in a vendor review, and they are not remotely the same exposure. This is the same least-privilege discipline security teams already apply to service accounts, extended down to the action level, with one difference: an agent can reason its way toward a capability nobody meant it to have, so the boundary has to sit at the tool call, not in the agent's assumed role.

Human Approval Gates

"Human in the loop" is one of the most common requirements in AI governance and one of the least specified. Runtime governance has to make it concrete: which actions need approval, who can approve them, what the approver actually sees, how long an approval stays valid, and how the decision gets logged.

What the approver sees is the part teams skip and later regret. An approval prompt that just says "the agent wants to send an email" gives a reviewer nothing to decide on, and over weeks it trains them to click approve without reading. A gate worth having shows the recipient, the data involved, and why the system flagged it, so the human is exercising judgment rather than rubber-stamping. Regulators have started naming this directly: oversight meant to counter a system's tendency toward automation bias doesn't do that job just by existing.

Approval should be risk-based, not universal. Review everything and the agent stops being useful. Review nothing and the policy is decoration. An agent might summarize internal notes without review and draft a CRM update without review, and still need a human before that update writes back or before a document leaves the company.

Context-Aware Decisions

The same action can be low risk or high risk depending on who's doing it, what data's involved, and where it's going. Posting in a test Slack channel isn't posting company-wide. Emailing a teammate isn't emailing a customer. A rule has to weigh several signals at once: agent identity, user role, data classification, destination, rather than a static role assignment, because a static rule can't tell the difference.

This is also what lets one rule scale across many agents instead of turning into a pile of agent-specific exceptions. Instead of writing "the support agent may not email customers," the organization writes "external sends of confidential data require approval," and that single rule now governs every agent that ever tries it.

Auditability

Agents need audit logs built for how they actually work. A useful trail captures who invoked the agent, which agent acted, what tools it called, what data it touched, which decisions applied, and what it ultimately did. That doesn't mean logging every prompt and every document forever; audit systems should capture references and decisions rather than copies of the sensitive material itself, wherever that's possible.

What it buys you shows up during an incident. When an agent does something unexpected, the question is whether the problem came from user intent, retrieved context, model output, a permission gap, or a missing approval, and each of those has a different fix. A prompt-injection vector in retrieved content gets fixed differently than a misconfigured permission, and a log that can't tell them apart leaves the team guessing.

Common Failure Modes

The same failure modes keep showing up, because they're the predictable result of governance sitting in a document instead of in the execution path.

• Privilege inheritance. An agent gets the full access of the user or service account it runs under, so it can reach far beyond its intended task. The policy may say least privilege; the deployment grants everything.
• Confused-deputy exposure. Untrusted content the agent retrieves, like an instruction hidden in a document, steers it into an action the user never asked for. Without action-level checks, the agent's legitimate permissions become the attacker's tool.
• Approval fatigue. Oversight is technically present but so frequent and so context-poor that reviewers approve by reflex. The gate exists in the architecture diagram and not in the actual decision.
• Attribution gaps. When something goes wrong, logs show a user or an application acting, with no record of which agent initiated the request or under whose delegation. The incident can't be reconstructed, so it can't be fixed with confidence.

None of this is exotic. Each one maps to exactly one of the capabilities above, which is a useful check on its own: if you can't say which capability would have caught a given failure, you probably don't have that capability yet.

Principles and Frameworks

A few design choices hold this together. The layer needs to work across whatever mix of models, agent frameworks, and SaaS tools different teams pick, without forcing every agent into one central runtime. Its primitives should be reusable, so a company isn't rebuilding identity, permission, and audit logic for every new agent. And it has to actually be enforceable at execution time, able to block, allow, escalate, redact, or log based on what it decides, not just describable in a document. None of this is worth much if it slows every project down with manual review; the goal is a safe default path, not a checkpoint.

None of this departs from the AI risk frameworks enterprises are already adopting; it gives them teeth. NIST's AI RMF organizes its work around four functions: govern, map, measure, and manage. The first three produce decisions and intentions, and they're closer to paperwork than engineering; a legal or compliance team can own most of that work directly. Manage, where risk actually gets responded to, is where runtime controls live, and no policy document performs that function on its own.

Regulation points the same direction. Under the EU AI Act, high-risk systems have to be designed for effective human oversight, proportionate to how autonomous the system is, and specifically built to counter the tendency to over-rely on a system's output. Writing "a human is in the loop" into a policy document satisfies none of that. A risk-based approval gate that actually surfaces context to a reviewer does. That's an engineering distinction more than a legal one: the requirement itself is legal, but meeting it takes a working approval gate, not a paragraph of legal text.

From Use Case to Infrastructure

This shows up across nearly every kind of agent deployment. A sales agent touches CRM data and drafts emails. A support agent reads tickets and escalates issues. A finance agent reads invoices and prepares approvals. An engineering agent inspects logs and opens pull requests. Each one mixes low-risk reads with higher-risk writes, on data of very different sensitivity, inside the same agent.

The finance agent that reads invoices all day is harmless until the moment it prepares a payment. The engineering agent summarizing incidents is low risk until it opens a pull request against production. What makes either one safe to run has nothing to do with the model. It comes down to whether the consequential step is gated and the routine ones aren't.

Enterprise security went through this exact shift already. Password policy alone gave way to identity systems that actually enforce it; trusting people to make good access calls gave way to role-based access control that doesn't depend on anyone remembering the rule. AI governance is repeating that pattern. The written rule won't disappear, but it becomes a specification, and the control built from it is the thing an auditor can actually point to.

Conclusion

The enterprises that get this right won't be the ones with the longest AI policy. They'll be the ones that can turn policy into something that runs: who's acting, on whose behalf, with which permissions, against which systems, and with what audit trail, answered at the moment an agent tries to act, not after.

That takes infrastructure: agent identity, granular permissions, human approval gates, and audit trails wired into the same path the agent actually uses. We don't think of these as separate products bolted on afterward. They're the connective tissue that lets an agent get deployed with the same confidence as any other system that touches production.