AI Agents and the EU AI Act: From Model Compliance to Runtime Governance

Overview

The EU AI Act sorts AI systems by the risk they create, with obligations that escalate sharply as the risk goes up. For most enterprises, the useful takeaway isn't that every AI tool gets treated the same way. It's that the organization needs a reliable way to know the risk profile of the systems it's actually running.

That gets harder with agents. An agent isn't a model behind a chat box. It's a model wired to tools, data sources, memory, and an execution environment, and it can operate inside one narrow workflow or coordinate across half a dozen systems. It might only recommend, or it might act directly.

Governing agents, as a result, means governing operational systems rather than isolated model calls. The questions shift:

• What is this agent designed to do, and who owns it?
• Which users can invoke it, and which tools and data can it reach?
• Which actions is it allowed to perform, and which need human approval?
• How are its actions logged and audited, and how can its access be modified or revoked?

Risk Tiers and Timeline

To apply the Act to a real deployment, start with how it sorts systems. Four tiers, and the obligations escalate sharply moving up:

• Unacceptable risk. A short list of practices banned outright under Article 5, including social scoring, certain manipulative techniques, untargeted facial-image scraping, and emotion recognition in workplaces and schools. Applicable since February 2, 2025.
• High risk. Systems used as safety components in regulated products, or that fall into sensitive Annex III uses such as employment, credit scoring, critical infrastructure, and education. The heaviest obligations: risk management, data governance, technical documentation, logging, human oversight, registration.
• Limited risk. Lighter transparency duties, such as disclosing that a person is talking to a chatbot or viewing AI-generated content.
• Minimal risk. The large remainder, not directly regulated by the Act.

The practical problem is that an agent rarely sits in one tier. A single workflow might combine a minimal-risk summarization step, a limited-risk customer message, and an action that touches a high-risk function such as an employment or credit decision. Classification isn't a property of the model. It's a property of what the agent is allowed to do at each step, and for whom.

The dates matter as much as the tiers. General-purpose model obligations became applicable August 2, 2025. Article 50's transparency duties, disclosing that a person is talking to an AI system and marking AI-generated content, apply from August 2, 2026, on schedule. The core high-risk package was originally due that same date; the Digital Omnibus on AI deferred it in June 2026, to December 2, 2027 for standalone Annex III systems and August 2, 2028 for high-risk AI embedded in already-regulated products. Several duties also fall on deployers rather than providers, including assigning competent human oversight and retaining logs for a set period, and for an enterprise running agents internally, that deployer role is usually the one that applies.

Not every one of those obligations changes what an engineering team builds. Some are paperwork the legal or compliance function owns outright:

The deferral moved the first two rows. It didn't move the last three, and those are the ones worth building before anyone asks.

Motivation

Early enterprise AI deployments were mostly advisory. A user asked, the model answered, a person decided what to do next. Even a bad answer had a limited blast radius, because the system wasn't acting on anything.

Agentic AI changes that. Once a model is wired to tools, its output can become an instruction, and once that instruction runs through an API, it can touch real business workflows: sending messages, updating records, retrieving documents, querying databases. None of that makes agents inherently unsafe. It does mean the enterprise has to govern the boundary between reasoning and action.

Take a sales agent connected to a CRM, email, Slack, and call notes. Asked to follow up with prospects who raised security concerns at a recent event, it might need to search meeting notes, identify accounts, read CRM fields, draft emails, and send some of them externally. Each step carries different risk. Searching approved event notes isn't the same as reading every customer record. Drafting an email isn't the same as sending it.

Figure 1 — One sales-agent workflow crossing three risk tiers. The classification is a property of the step, not of the agent or the model behind it.

These boundaries erode quietly, too. An agent scoped to draft emails picks up one new tool during a demo. A pilot connector keeps its production credentials past the pilot. A prompt-injected document talks the agent into exfiltrating data it technically had permission to read. We don't think any of that is exotic; it's the ordinary drift of a system that acts, and governing the reasoning-to-action boundary is what keeps the drift from becoming an incident.

This is where most enterprise AI programs will struggle. Humans authenticate through identity providers. Applications get API credentials. SaaS tools enforce role-based permissions. Agents sit between those categories: they act on behalf of users, operate through applications, and decide using model-driven reasoning, and without a dedicated governance layer their access is hard to reason about.

From Model Governance to Agent Governance

Model governance is about the model itself: performance, data provenance, evaluation, bias, safety, vendor management. Still foundational, especially in regulated contexts. Agent governance is a separate layer on top of it: what the system can do once that model is wired to tools and enterprise context, including its identity, permissions, accessible systems, allowed actions, and audit trail.

The distinction matters because two agents built on the same model can carry very different risk. A documentation assistant answering from public help-center content is low risk. A finance agent that reads invoices, updates payment status, and initiates approval workflows is not, even on an identical model. The surrounding permissions and actions create the governance requirement, not the model.

• Model governance asks: which model, who provides it, how does it perform, what are its known failure modes?
• Agent governance asks: what is this agent for, who owns it, which tools and data can it reach, which actions need approval, how is each one recorded?

This matters for compliance specifically. If a company can't say what an agent was allowed to access, what action it attempted, and who approved it, it will struggle to demonstrate real control over the system. The Act's human-oversight and record-keeping requirements aren't satisfied by a strong model. They're satisfied by the operational layer built around it.

Architectural Requirements for EU AI Act Readiness

The Act should push enterprises toward a more operational approach to governance: infrastructure that can enforce decisions at runtime, not only policy that lives in a review board's minutes. The requirements below are framework-agnostic. They apply whether agents run on an orchestration library, a hosted assistant API, or an internal platform.

1. Agent Inventory

The first requirement is visibility: which agents exist, who owns them, what they're for. This is more than a list of AI tools. An inventory should capture each agent's connected systems, user scope, data access, allowed actions, approval requirements, and lifecycle state, structured enough to be queried and tied to the agent's actual runtime configuration rather than sitting in a spreadsheet:

{
  "agent_id": "sales-followup-agent",
  "owner": "revenue-ops",
  "purpose": "Draft follow-up emails for event prospects",
  "model": "provider/model-name",
  "risk_tier": "limited",          // customer-facing output
  "allowed_tools": ["crm.read", "notes.search", "email.draft"],
  "blocked_tools": ["email.send", "crm.update"],
  "approval_required": ["email.send"],
  "data_scope": "accounts owned by invoking user",
  "lifecycle": "active",
  "owner_contact": "revenue-ops@example.com"
}

Without an inventory, a team can deploy a new agent without security review because it looks like a simple productivity tool, even with access to sensitive customer or employee data. An unmanaged agent can keep running with broad access long after the experiment that justified it has ended.

2. Agent Identity

Each agent needs an identity distinct from the human user, the model, and the application it operates through, one that defines what the agent is, who owns it, and which users can invoke it. Without that, actions become hard to attribute: a log shows a user updated a record, when an agent performed the update on the user's behalf, or shows an application calling an API with no indication of which agent initiated it or why.

If an agent takes an unexpected action, the organization needs to know whether it was authorized, which rule applied, whether a human approved it, and how to prevent a repeat. Every action should carry both the agent's identity and the identity of the user on whose behalf it acts, so attribution never collapses into one ambiguous service account.

Agent identity is the foundation for accountability.

3. Permission Boundaries

Agents should operate under explicit boundaries defining not just which systems they can reach, but which actions they can take inside them. Traditional OAuth scopes and SaaS permissions are usually too coarse for this: an agent may need to draft an email but not send it, read selected CRM fields but not update opportunities, search approved documents but not access every file the user can see.

Least privilege for agents means action-level thinking, not mirroring a user's full access. Expressed as policy, that looks less like a single scope and more like a per-action rule:

# Permission policy for sales-followup-agent
allow:  notes.search       when source == "approved-event-notes"
allow:  crm.read           when field in ["name", "stage", "last_contact"]
allow:  email.draft        always
deny:   crm.update         always
review: email.send         when recipient.domain != "example.com"

The exact syntax matters less than the granularity: each rule ties an action to the conditions under which it's permitted, denied, or escalated for review.

4. Human Approval and Oversight

The Act treats human oversight as a concrete requirement, not a slogan: high-risk systems must be designed so assigned people can understand the system's limitations, monitor its operation, stay alert to automation bias, and decide not to act on its output when warranted. In agentic systems that means defining which actions need approval, who can approve them, what the approver sees, and how the decision is recorded.

Approval should scale with risk. Summarizing approved internal documentation may not need review. Sending external communications, changing customer records, exporting data, or triggering financial workflows usually should. And the quality of the oversight depends entirely on what the approver actually sees: approving an action without the underlying data, the agent's stated intent, and the affected records is a rubber stamp, not oversight.

5. Runtime Permission Enforcement

Permissions have to be enforceable while an agent is running, not only describable in documentation that never touches a system acting dynamically across tools.

This is where the enterprise evaluates each attempted action in context. The same action might be allowed in one situation, blocked in another, and need approval in a third: a Slack message to a private channel is low risk, the same message to a company-wide channel or an external partner may need review; reading a public document is fine, retrieving sensitive customer data may need an extra check.

The architectural point is that the check belongs where the agent calls a tool, not only in the prompt. A model can be instructed to behave, but instructions are advisory; an authorization check at the tool boundary is dispositive. Put the decision point between the agent and the action, and context matters the way it should, because the risk depends on the user, the agent, the tool, the data, and the action attempted.

6. Auditability

A useful audit trail captures the user request, the agent identity, the tools called, the data sources accessed, the decisions applied, any human approvals, and the final action taken. This lines up with the Act's record-keeping expectations, which require high-risk systems to log events automatically so their operation can be traced, and which oblige deployers to retain those logs for a defined period.

The goal isn't storing unlimited sensitive content; logs should avoid exposing secrets, credentials, or unnecessary personal data. But the organization needs enough to reconstruct what happened during an incident, a compliance review, or an operational failure. Auditability is what turns agentic systems from opaque automation into infrastructure someone can actually be held accountable for.

Strategic Implications

The Act is part of a broader shift toward AI accountability, and enterprises will increasingly be expected to prove control over the systems they deploy, not just document intent to control them. For agentic AI, that means more than model documentation and acceptable-use policy. It means operational infrastructure for managing agents in production.

That creates an opening for companies that invest early. A business team can ship a new agent faster once the approval path and permission model already exist instead of getting invented from scratch each time. Security stops being the function that says no by default, because it can actually see what it's approving. And compliance stops treating governance as a blocker bolted on at the end, since agent behavior that's already logged and attributable is most of the evidence they'd otherwise have to go chase down.

There's a portability argument too. The Act is the most developed regime today, but not the only one taking shape. We think the controls it implies (inventory, identity, least privilege, oversight, enforcement, audit) are close to what most emerging frameworks and internal security reviews will end up asking for. Building that layer once, around what agents actually do, tends to generalize across jurisdictions better than chasing each new regulation's specific text.

Toward Production-Ready Agentic AI

The future of enterprise AI won't be defined only by better models. It will be defined by the systems that make models usable in real organizations: identity, permissions, connectivity, approvals, and observability.

The Act reinforces that shift without being the whole reason for it. As AI systems become more capable and more embedded in business workflows, enterprises will need to show they understand what those systems are doing and that they can control them.

For AI agents, the central governance question is simple: can the organization explain and enforce what each agent is allowed to do? If the answer is no, the deferred deadline doesn't make the agent production-ready. The work that was always going to matter was never on the high-risk clock to begin with.