From One Agent Runtime to Many: What Actually Needs to Stay Centralized

TL;DR

"Decentralize your agent architecture" is usually bad advice stated at the wrong altitude. The layer that should fragment across teams is execution, which agent framework runs the reasoning loop. The layers that should stay centralized are connectivity and identity, how an agent finds a tool and proves who it's acting for, because those need one source of truth to avoid getting reimplemented, and re-scoped, slightly wrong by every team that touches them.

We think most of the pain attributed to "centralization" is actually one team's execution runtime becoming every other team's bottleneck, not centralization itself being the wrong idea. The fix isn't tearing down the central service. It's being precise about which parts of it were never supposed to be a bottleneck in the first place.

Decentralizing everything, execution, connectivity, credentials, logging, trades a single throat to choke for duplicated integrations, inconsistent scoping, and an audit trail that stops covering the whole system. That trade only pays off once an organization has enough independently built agents that one team genuinely can't review them all.

If you're deciding this for three agents built by one team, you don't have the problem this architecture solves yet.

Overview

"Centralized" and "decentralized" get used as if they describe one axis, when an agent stack actually has at least three layers that can each be centralized or not independently: where the agent's reasoning runs, how it reaches tools and other agents, and how its behavior gets observed and governed. Treating all three as one decision is how organizations end up either locked into a single team's execution roadmap, or with a dozen teams each reinventing credential storage badly.

The useful question isn't "should we decentralize." It's which of these three layers benefits from letting teams move independently, and which one gets worse the moment there's more than one of it. Those turn out to have different answers, and the answer for connectivity is the one that surprises people who came in expecting this post to argue for decentralizing everything.

Where One Runtime Breaks Down

Take an organization that stood up one central service to run agents: one framework, one team operating it, one review queue for anything new. It works well with two agents built by the team that owns the service. It stops working somewhere around the point where five other teams each want an agent of their own, because now every new tool, every framework upgrade, and every review of a proposed agent competes for the same team's time. The central team didn't get worse at their job. The number of people depending on their job grew faster than they did.

That's the actual failure mode, and it's worth naming precisely because the fix people reach for is usually too broad. The problem was never that one team ran the reasoning loop for every agent in the company. The problem was that the reasoning loop, the tool connections, and the audit trail were all bottlenecked on the same team at once, so the fix that actually works is letting the part that was genuinely a bottleneck, who builds and runs the agent logic, spread out, while keeping the parts that need one consistent answer, how an agent gets a credential and what gets logged when it acts, where they were.

The Execution Layer

This is where the agent's reasoning actually runs: the framework, the prompting approach, whichever pattern from planner-executor to a simple tool-calling loop the team building the agent has decided fits their task. It's also the layer that churns fastest. Frameworks get replaced, teams have real opinions about which one suits their workload, and a mandate to use one specific runtime company-wide ages badly within a couple of product cycles.

This is the layer worth decentralizing without much argument. Let a support team build their agent in whatever framework their engineers already know. Let a data team run something code-first while a business team uses a visual builder. None of that creates the coordination problem that made the original centralized runtime painful, because none of it requires a shared source of truth the way credentials or audit logs do. What it does require is that every one of those execution environments plugs into the same connectivity and identity layer underneath it, which is the part worth protecting.

The Connectivity Layer

This is the layer most people mean when they say "decentralize," and it's the one we'd argue against decentralizing at all. Connectivity is how an agent discovers a tool, authenticates as itself, and gets scoped down to only what this specific task needs. Let every execution team stand up its own version of that, and you get the same OAuth flow implemented five times with five different bugs, the same customer database connected through five different credentials with five different blast radii, and no single place to answer "what can this agent actually touch."

Take a finance team and a support team that each build an agent needing access to the same billing system. Two credentials, independently provisioned, independently scoped, is worse than one shared connection with two sets of permissions layered on top of it, because the moment one of those credentials needs to be revoked or audited, someone has to first figure out that a second one exists. Centralizing connectivity isn't about control for its own sake. It's about there being exactly one place where "which agents can reach this system, and with what scope" has a single, checkable answer.

The Operations Layer

Logging, tracing, and evaluation face the same argument as connectivity, for a related reason. The entire point of an audit trail is being able to answer a question across every agent at once, which incident happened, which team's agent caused it, whether the same failure is showing up somewhere else. Split that across five teams' own logging setups and you've kept the label "observability" while losing the one property that made it useful: a single place to look when something goes wrong and you don't yet know which agent is responsible.

Evaluation has the weaker version of the same problem. A team grading its own agent against its own bar tends to grade generously, and there's no way to compare two teams' agents against each other if they were never measured the same way. None of this means every team needs to file a ticket with a central group to check a dashboard. It means the schema for what gets logged and how success gets measured has to be shared, even while the teams building the agents that produce those logs stay independent.

What Decentralizing Actually Costs

None of this is free, in either direction, and the piece usually left out of an architecture pitch is the cost side. A single centralized runtime is genuinely simpler to reason about: one team, one review process, one place to look when something breaks. Splitting execution across teams buys speed and autonomy at the cost of that simplicity, more teams making independent decisions about frameworks and patterns, more surface area for someone to build something that works but doesn't fit the shared connectivity layer cleanly.

That trade is worth making once an organization has enough independently built agents that a single team reviewing all of them has become the actual constraint on shipping anything. It is not worth making for an early-stage effort with one or two agents built by one team; that team doesn't have a bottleneck problem yet, and the coordination overhead of splitting execution out prematurely is pure cost with no offsetting benefit. Build the centralized version first. Split execution out when the review queue, not the architecture diagram, tells you it's time.

Conclusion

Shared connectivity + identity --> Shared operations Team C agent --/ (one credential store, (one log schema, one scope model) one eval bar) Execution fragments across teams and frameworks. Connectivity and operations don't. -->

Figure 1 — Decentralize where teams need independence: execution. Keep one shared answer where consistency is the point: connectivity, identity, and how behavior gets logged and measured.

The advice worth taking from "centralized versus decentralized" isn't a single answer applied to the whole stack. It's a per-layer decision, and the layers don't all point the same direction. Let execution fragment, because that's where a shared mandate turns into a bottleneck and where teams have legitimate reasons to disagree. Keep connectivity, identity, and operations centralized, because those are exactly the places where having five inconsistent answers is worse than having one team own the answer, even an imperfect one.

Getting this wrong in either direction has a predictable shape. Centralize everything and one team becomes the approval queue for the whole company's agent roadmap. Decentralize everything and you get five credential stores, five scoping models, and an audit trail that only covers whichever team remembered to build one. The organizations that get this right split the decision by layer, not by allegiance to one side of the debate.