AI Agents and the EU AI Act: From Model Compliance to Runtime Governance

As enterprises move from experimental AI assistants to production-grade agentic systems, AI governance must evolve beyond model selection, vendor review, and acceptable-use policies. The next generation of enterprise AI systems will not simply generate text or summarize documents. They will retrieve data from internal systems, call APIs, execute workflows, interact with users, and operate across multiple applications on behalf of employees, customers, and business functions.

This shift creates a new compliance and governance challenge. Traditional AI governance frameworks often focus on the model: how it was trained, what data it uses, how it performs, and whether it introduces bias, safety, privacy, or security risks. These questions remain important, but they are incomplete for agentic systems. Once an AI system can take action, the enterprise must also govern the runtime environment around the model: identity, permissions, tool access, human approval, logging, policy enforcement, and lifecycle management.

The EU AI Act is one of the clearest signals that AI governance is becoming a core enterprise responsibility. While much of the discussion around the Act focuses on model providers and high-risk AI systems, the operational implications are broader. Enterprises deploying AI agents will need to understand not only which models they use, but also how those models are embedded into business workflows, what systems they can access, what actions they can perform, and how those actions are controlled.

It also matters that the Act is no longer a future hypothetical. Its prohibitions on unacceptable-risk practices and its AI literacy obligation became applicable on February 2, 2025, and the obligations for general-purpose AI models followed on August 2, 2025. The bulk of the high-risk and transparency requirements apply from August 2, 2026, with the broader rollout extending into 2027 and beyond for AI embedded in regulated products. For enterprises, that timeline means the question is no longer whether to prepare, but whether the systems already in production can be explained and controlled.

Overview

The EU AI Act establishes a risk-based framework for AI systems. Its goal is to regulate AI based on the level of risk the system creates, with stricter obligations for uses that may affect safety, fundamental rights, employment, education, law enforcement, critical infrastructure, and other sensitive domains.

For enterprises, the most important takeaway is not that every internal AI tool will be treated the same way. Rather, organizations need a reliable way to understand the risk profile of the AI systems they deploy. That requires visibility into where AI is used, what purpose it serves, what data it touches, and what business outcomes it affects.

This becomes more complicated with agents. An agent is not just a model exposed through a chat interface. It is a model connected to tools, instructions, memory, data sources, policies, and execution environments. The agent may operate within a narrow workflow, or it may coordinate across multiple systems. It may provide recommendations, or it may directly perform actions. It may be invoked by a single user, or it may run as part of an automated process.

As a result, enterprises need to govern agents as operational systems rather than isolated model calls. The key questions shift from model-centric governance to system-level governance:

• What is this agent designed to do?
• Who owns it?
• Which users can invoke it?
• Which tools and data sources can it access?
• What actions is it allowed to perform?
• Which actions require human approval?
• How are its actions logged and audited?
• How can its access be modified, paused, or revoked?

These questions are central to enterprise AI governance because they determine how much control the organization actually has over agentic behavior in production.

Risk Tiers and Timeline

To apply the Act to a real deployment, it helps to start with how it sorts systems. The framework defines four broad levels of risk, and the obligations escalate sharply as you move up the pyramid:

• Unacceptable risk. A short list of practices is banned outright under Article 5, including social scoring, certain manipulative or exploitative techniques, untargeted scraping of facial images, and emotion recognition in workplaces and schools. These obligations have applied since February 2, 2025.
• High risk. Systems used as safety components in regulated products, or that fall into the sensitive use cases listed in Annex III such as employment, credit scoring, critical infrastructure, and education. These carry the heaviest obligations: risk management, data governance, technical documentation, logging, human oversight, and registration.
• Limited risk. Systems subject to lighter transparency duties, such as making it clear to people when they are interacting with a chatbot or viewing AI-generated content.
• Minimal risk. The large remainder of applications, which the Act does not regulate directly.

The practical difficulty is that an agent rarely sits in one tier. A single agentic workflow might combine a minimal-risk summarization step, a limited-risk customer-facing message, and an action that touches a high-risk function such as an employment or credit decision. The risk classification is therefore not a property of the model. It is a property of what the agent is allowed to do at each step, and for whom.

The timeline matters just as much as the tiers. Obligations for general-purpose AI models became applicable on August 2, 2025. The core requirements for high-risk systems and the transparency rules apply from August 2, 2026, with the wider rollout continuing through August 2, 2027 and an extended transition for high-risk AI embedded in already-regulated products. Several obligations also fall directly on deployers rather than providers, including assigning competent human oversight and retaining system logs for a defined retention period. For an enterprise running agents internally, that deployer role is usually the one that applies, which is precisely why runtime controls and audit trails are not optional.

Motivation

Early enterprise AI deployments were often low-risk because they were advisory. A user asked a question, the model generated an answer, and a human decided what to do next. Even when the output was imperfect, the blast radius was usually limited because the AI system did not directly act on enterprise systems.

Agentic AI changes this risk model. When a model is connected to tools, its output can become an instruction. When that instruction is executed through an API, the AI system can affect real business workflows. It can send messages, update records, retrieve documents, create tickets, trigger automations, query databases, or interact with customers.

This does not make agents inherently unsafe. It does mean that the enterprise must govern the boundary between reasoning and action.

Consider a sales agent connected to a CRM, email, Slack, and internal call notes. A user may ask the agent to follow up with prospects from a recent event who mentioned security concerns. To complete the request, the agent may need to search meeting notes, identify relevant accounts, read CRM fields, draft personalized emails, and potentially send those emails externally. Each of these steps carries a different level of risk. Searching approved event notes is very different from reading all customer records. Drafting an email is different from sending it. Summarizing internal security concerns is different from including those concerns in an external message.

The governance problem is not simply whether the agent can produce a good answer. The problem is whether the system can enforce the correct boundaries as the agent moves through the workflow.

These boundaries also have a way of eroding quietly. An agent that was scoped to draft emails gets one new tool added during a demo. A connector that was meant for a pilot keeps its production credentials. A prompt-injected document convinces the agent to exfiltrate data it technically had permission to read. None of these are exotic failures; they are the ordinary drift of a system that acts. Governing the reasoning-to-action boundary is what keeps that drift from becoming an incident.

This is where many enterprise AI programs will struggle. Existing controls are often designed for either human users or traditional applications. Humans authenticate through identity providers. Applications receive API credentials. SaaS tools enforce role-based permissions. But agents sit between these categories. They act on behalf of users, operate through applications, and make decisions using model-driven reasoning. Without a dedicated governance layer, their access can become difficult to reason about.

From Model Governance to Agent Governance

Model governance focuses on the behavior and risk of the underlying AI model. It includes questions such as model performance, data provenance, evaluation, bias, safety, security, and vendor management. These remain foundational concerns, especially for organizations deploying AI in regulated or high-impact contexts.

Agent governance adds another layer. It focuses on what the AI system can do once the model is connected to tools and enterprise context. This includes the agent’s identity, permissions, accessible systems, allowed actions, approval requirements, audit logs, and lifecycle state.

The distinction matters because two agents using the same model can have very different risk profiles. A documentation assistant that answers questions from public help-center content is relatively low risk. A finance agent that can read invoices, update payment statuses, and initiate approval workflows is materially different. The model may be identical, but the surrounding permissions, data, and actions create a very different governance requirement.

For this reason, enterprises should avoid treating AI governance as a purely model-level review. A model registry alone is not enough. Organizations also need an agent registry, a tool registry, and a clear map of the relationships between users, agents, tools, data sources, and actions.

A simple way to make the distinction concrete is to write down what each layer is accountable for. The model layer answers questions about how the system reasons. The agent layer answers questions about what the system is permitted to do with that reasoning:

• Model governance asks: Which model is this, who provides it, how does it perform, and what are its known failure modes?
• Agent governance asks: What is this agent for, who owns it, which tools and data can it reach, which actions need approval, and how is each action recorded?

This is especially important for compliance. If a company cannot explain what an agent was allowed to access, what action it attempted, which policy decision was applied, and who approved the action, it will be difficult to demonstrate meaningful control over the system. The Act’s requirements around human oversight and record-keeping are not satisfied by a strong model alone; they are satisfied by the operational layer that sits around it.

Architectural Requirements for EU AI Act Readiness

The EU AI Act should push enterprises toward a more operational approach to AI governance. Rather than relying only on policies and review boards, organizations need infrastructure that can enforce governance decisions at runtime. The requirements below are deliberately framework-agnostic: they apply whether agents are built on an orchestration library, a hosted assistant API, or an internal platform.

1. Agent Inventory

The first requirement is visibility. Enterprises need to know which agents exist across the organization, who owns them, and what purpose they serve. This is more than a list of AI tools. An agent inventory should capture the operational characteristics of each agent, including connected systems, user scope, data access, allowed actions, approval requirements, and lifecycle state.

Without an inventory, organizations cannot reliably classify AI systems, assess risk, enforce policies, or respond to incidents. An unmanaged agent may continue operating with broad access long after the original experiment has ended. A team may deploy a new agent without security review because it appears to be a simple productivity tool, even though it has access to sensitive customer or employee data.

An agent inventory creates the foundation for governance because it turns invisible AI usage into a manageable set of systems. In practice, the inventory works best as a structured record rather than a spreadsheet, so it can be queried, versioned, and tied to the agent’s actual runtime configuration:

{
  "agent_id": "sales-followup-agent",
  "owner": "revenue-ops",
  "purpose": "Draft follow-up emails for event prospects",
  "model": "provider/model-name",
  "risk_tier": "limited",          // customer-facing output
  "allowed_tools": ["crm.read", "notes.search", "email.draft"],
  "blocked_tools": ["email.send", "crm.update"],
  "approval_required": ["email.send"],
  "data_scope": "accounts owned by invoking user",
  "lifecycle": "active",
  "owner_contact": "revenue-ops@example.com"
}

A record like this turns an abstract governance question into something an auditor, a security engineer, and the agent runtime can all read the same way.

2. Agent Identity

Each agent should have a distinct identity separate from the human user, the underlying model, and the application through which it operates. This identity should define what the agent is, who owns it, what it is designed to do, which users can invoke it, and what systems it can access.

Without agent identity, actions become difficult to attribute. A log may show that a user updated a record, when in reality the update was performed by an agent acting on the user’s behalf. Another log may show that an application called an API, without indicating which agent initiated the action or why.

This ambiguity creates problems for security, compliance, and incident response. If an agent takes an unexpected action, the organization needs to know whether the action was authorized, which policy applied, whether a human approved it, and how to prevent similar behavior in the future. A useful model is to think of every action as carrying both the agent’s identity and the identity of the user on whose behalf it acts, so attribution never collapses into a single ambiguous service account.

Agent identity is the foundation for accountability.

3. Permission Boundaries

Agents should operate under explicit permission boundaries. These boundaries should define not only which systems an agent can access, but also which actions it can perform within those systems.

Traditional OAuth scopes and SaaS permissions are often too coarse for agentic workflows. An agent may need to draft an email but not send it. It may need to read selected CRM fields but not update opportunities. It may need to search approved documents but not access every file available to the user. It may need to create a support ticket but not close or escalate one.

Least privilege for agents requires action-level thinking. The goal is not simply to mirror a user’s full access. The goal is to grant the agent only the capabilities required for its intended workflow, with additional controls for sensitive actions. Expressed as policy, that looks less like a single scope and more like a per-action decision:

# Permission policy for sales-followup-agent
allow:  notes.search       when source == "approved-event-notes"
allow:  crm.read           when field in ["name", "stage", "last_contact"]
allow:  email.draft        always
deny:   crm.update         always
review: email.send         when recipient.domain != "example.com"

The point is not the exact syntax but the granularity: each rule binds an action to the conditions under which it is permitted, denied, or escalated for review.

4. Human Approval and Oversight

Human oversight is a central concept in AI governance, and the EU AI Act treats it as a concrete requirement rather than a slogan: high-risk systems must be designed so that assigned people can understand the system’s limitations, monitor its operation, stay alert to automation bias, and decide not to act on its output when appropriate. Translating that into agentic systems means defining which actions require approval, who is allowed to approve them, what information the approver sees, and how the approval is recorded.

Approval requirements should be based on risk. Low-risk actions, such as summarizing approved internal documentation, may not need review. Higher-risk actions, such as sending external communications, changing customer records, exporting data, modifying access permissions, or triggering financial workflows, may require explicit approval.

The quality of oversight also depends on what the approver actually sees. Approving an action without seeing the underlying data, the agent’s stated intent, and the affected records is a rubber stamp, not oversight. Effective oversight should preserve the productivity benefits of agents while ensuring that sensitive actions remain accountable, with enough context at the approval step that a human can meaningfully say no.

5. Runtime Policy Enforcement

Policies must be enforceable during agent execution. A governance framework that exists only in documentation cannot reliably control a system that acts dynamically across tools.

Runtime policy enforcement allows the enterprise to evaluate each attempted action in context. The same action may be allowed in one situation, blocked in another, and require approval in a third. For example, sending a Slack message to a private internal channel may be low risk, while posting to a company-wide channel or messaging an external partner may require review. Reading a public document may be allowed, while retrieving sensitive customer data may require additional checks.

The architectural implication is that enforcement belongs at the point where the agent calls a tool, not only in the prompt. A model can be instructed to behave, but instructions are advisory; a policy check at the tool boundary is dispositive. Placing the decision point between the agent and the action is what makes context-aware enforcement reliable, because the risk depends on the user, the agent, the tool, the data, and the action being attempted.

6. Auditability

Enterprises need audit logs designed for agent behavior. A useful audit trail should capture the user request, the agent identity, the tools called, the data sources accessed, the policy decisions applied, any human approvals, and the final action taken. This aligns directly with the Act’s record-keeping expectations, which require high-risk systems to log events automatically so that their operation can be traced, and which oblige deployers to retain those logs for a defined period.

The goal is not to store unlimited sensitive content. Logs should be designed carefully to avoid exposing secrets, credentials, or unnecessary personal data. However, the organization must retain enough information to reconstruct what happened during an incident, compliance review, or operational failure.

Auditability turns agentic systems from opaque automation into accountable infrastructure.

Strategic Implications

The EU AI Act is part of a broader shift toward AI accountability. Enterprises will increasingly be expected to understand, govern, and prove control over the AI systems they deploy. For agentic AI, this means organizations need more than model documentation and acceptable-use policies. They need operational infrastructure for managing agents in production.

This creates a strategic opportunity for companies that invest early in agent governance. Instead of treating compliance as a blocker, they can use governance as an adoption enabler. Business teams can deploy agents faster when there are clear approval paths, reusable permission models, and shared security patterns. Security teams can support AI adoption more confidently when they have visibility and control. Compliance teams can evaluate systems more effectively when agent behavior is logged and attributable.

There is also a portability argument. The EU AI Act is the most developed regime today, but it is not the only one taking shape, and the controls it implies — inventory, identity, least privilege, oversight, enforcement, and audit — are the same controls that satisfy most emerging frameworks and internal security reviews. Building the governance layer once, around what agents do, tends to generalize across jurisdictions far better than chasing the specific text of each new regulation.

Toward Production-Ready Agentic AI

The future of enterprise AI will not be defined only by better models. It will be defined by the systems that make models usable in real organizations: identity, permissions, connectivity, approvals, observability, evaluation, and governance.

The EU AI Act reinforces the need for this shift. As AI systems become more capable and more embedded in business workflows, enterprises will need to demonstrate that they understand what those systems are doing and that they can control them appropriately.

For AI agents, the central governance question is simple: can the organization explain and enforce what each agent is allowed to do?

If the answer is no, the agent is not truly production-ready.

Agentic AI requires a new operational layer: one that treats agents as first-class enterprise actors with identities, permissions, lifecycle states, and audit trails. That layer is what allows organizations to move from experimentation to deployment, from informal policy to runtime control, and from isolated AI tools to governed agentic ecosystems.