Rowboat realism marine painting
← Back to blog

ENGINEERING

A Developer's Guide to Thinking in Agents, Not Apps

An app is safe to get wrong because a human reviews the output before anything happens. An agent removes that review by design — here's what that costs, concretely.

Paulina XuApr 13, 20269 min
EngineeringMental ModelsAgents

TL;DR

An app is safe to get wrong because a human reads the output before anything happens. An agent is not, because the output is the action. That single difference is the whole reason agent code that looks like app code keeps producing incidents nobody can quite explain.

Take a scheduling assistant that used to suggest three open slots for a person to pick from. Someone adds one line of scope: let it book the meeting once it finds a slot. The code barely changed. What changed is that a wrong answer used to be a UI bug, and now it's a calendar invite that already went out.

Teams read that as a reliability problem and reach for a bigger model. It is usually a design problem: the system was built assuming a human stood between the decision and the consequence, and nobody removed that assumption when they removed the human.

We think the fix isn't a framework. It's deciding, tool by tool and action by action, which ones still get a human in front of them and which ones don't — before the agent ever runs, not after it acts.

Overview

Take a scheduling assistant built the way most internal tools get built. A request comes in with an email address and a rough time window, a handler checks the calendar for conflicts, and it hands back a short list of times that work. That's a good, boring app. Then someone extends its scope by one sentence: once it finds a slot, book it. The code barely changes. What changes is that the same logic that used to produce a suggestion now produces a sent invite, and a bug that used to be a wrong recommendation on a screen is now a meeting on somebody's calendar that never should have existed.

That is the shift this post is about. Not a bigger model, not a different framework. An app is designed on the assumption that a person looks at the output before anything real happens: they read the recommendation, approve the transaction, click send. An agent, by definition, is the thing standing where that person used to stand. Every assumption baked into app-shaped code about what a human will catch has to be re-earned, or it quietly becomes a hole.

The frustrating part is that the symptoms hide the cause. When an agent sends the wrong thing, calls the wrong tool, or loops on a task it should have abandoned, the instinct is to reach for a smarter model. Most of the time the model reasoned correctly from what it had. The system around it just never asked whether this particular action was one a human needed to see first.

The question an app never has to answer is the one an agent can't avoid: who looks at this before it happens, and what happens if nobody does? Everything below is a different angle on that same question.

A Loop That Acts, Not a Function That Returns

An app's unit of work is a request and a response. Something calls it, it does a bounded amount of work, and it hands back a value that somebody or something else decides what to do with. An agent's unit of work is a loop: observe the state, decide what it means for the goal, take an action, fold the result back in, and decide whether to continue. The loop doesn't hand a decision back for review. It contains the decision.

while (!done && steps < maxSteps) {
  const observation = perceive(state);
  const thought = reason(goal, observation);
  const action = decide(thought);
  const result = act(action);          // this line is where an app stops and an agent doesn't
  state = update(state, result);
  done = isGoalMet(state);
}

The line worth staring at is act(action). In an app, the equivalent line returns a value to a caller. In an agent, it runs in the real world: it sends the email, writes the row, calls the refund API. Everything upstream of that line, perceiving, reasoning, deciding, can be exactly as careful as the best app code you've ever written and it still won't matter if nothing checks what happens the moment act actually executes.

Tools Are Choices the Agent Makes, Not Calls You Schedule

In an app, a function call happens at a place in the code you chose. In an agent, a tool is a capability the model decides whether to reach for, based on nothing but its description and the situation in front of it. That description is now part of the runtime, not documentation sitting next to the code.

// Vague: the agent has to guess when this applies, and it will guess wrong
{ name: "get_data", description: "Gets data." }

// Precise: the agent knows the precondition, the output, and the failure behavior
{
  name: "lookup_customer_by_email",
  description: "Returns a customer's account record given their email. Use when you need " +
    "order history, plan tier, or billing status. Returns null if no account matches; do not retry on null.",
}

The second version is doing work that, in an app, would have lived in an if statement a developer wrote and reviewed once. Here it lives in a sentence the model reads fresh on every decision. Naming a tool for exactly what it does, stating what it returns on failure, and saying explicitly what not to do with that failure is the agent equivalent of writing the branch you used to write in code. Skip it, and the model writes that branch itself, silently, and you find out what it decided after the fact.

An Exit, Not Just a Path

Two lines in the loop above do more work than the rest combined, and both are things an app never needs. maxSteps is what stops a confused agent from spinning forever on a task it can't complete. isGoalMet is what lets it admit it failed instead of returning something confident and wrong so the loop can end. An agent that can't stop and an agent that can't say it didn't finish are the same failure wearing different clothes.

Retries follow the same logic. An app's retry replays an identical request and hopes the network cooperates this time. An agent's retry should almost never be identical: the tool returned an empty result, so it broadens the query; the structured output didn't parse, so it reformats; the written record doesn't match what it expected, so it re-reads before trying again. A retry that just repeats the same call with the same arguments is a bet that the world changed on its own. Most of the time it didn't, and the agent burns a budget of attempts finding that out the hard way.

Neither of these is optional once the loop can act. A maxSteps guard on a function that only returns a value is a nice-to-have. On a loop with a live act() call inside it, it's the difference between a task that times out and one that keeps sending the same email until someone notices.

What Acting on Its Own Actually Costs You

This is the part app-shaped thinking gets wrong in a way that only shows up once the agent is live. In an app, a mistake is a bad output waiting for someone to notice it before it matters. In an agent, a mistake is already an action, and by the time anyone reviews it, the review is a postmortem, not a gate.

Concretely: the scheduling assistant that now books meetings has to answer a question the suggestion-only version never faced. What if the slot it picked is technically free but obviously wrong, a 2 a.m. call across nine time zones that a scheduling API has no way to flag? An app returning three options lets a human catch that instantly. An agent that books the meeting has already sent the invite before anyone sees it. The fix is not a smarter model that never picks a bad slot. It's deciding, before the agent runs, that booking is an action a human confirms and suggesting is not, and building that distinction into the system rather than hoping the model infers it.

That distinction has to be made tool by tool. Reading a calendar, drafting a message, and proposing a transaction are cheap to get wrong, because a human still stands between the draft and the consequence. Sending the message, executing the transaction, and deleting the row are not, because the agent's decision and the world's state change at the same moment. An app-shaped design treats all of these the same, because in an app they mostly are the same: outputs waiting for someone to act on them. An agent-shaped design has to sort them by what happens if nobody's watching when the action fires, and route the second group through a confirmation the first group doesn't need.

agent

too late

goal

loop:
observe / reason / decide

act()

real world changes
(sent, written, executed)

human finds out

app

request

handler

output

human reviews,
then acts

human finds out The review point moves from before the consequence to after it, unless you deliberately put it back. -->

Figure 1 — Where the human review point sits in each design. An app puts it before the consequence by default. An agent puts it after, unless the boundary is built in on purpose.

Governing an agent means deciding, in advance, which actions get a human in front of them and which don't. Not because the model can't be trusted, but because "acted, and we found out later" is the wrong shape of feedback loop for anything you can't undo.

Conclusion

None of this argues against building agents, or for keeping a human in every loop until the whole idea stops paying for itself. It argues for noticing where the human review that made app-shaped code safe quietly went missing, and putting a deliberate boundary back in its place: a step budget so the loop can't spin forever, an honest failure signal so it can admit defeat, tool descriptions precise enough that the model's choices are predictable, and a clear, tool-by-tool line between actions that can run on their own and actions that need a person to see them first.

We'd start that line with reversibility. If an action can be cleanly undone, let the agent take it. If it can't, the agent proposes and a person decides, every time, until you have enough evidence about that specific action to trust it otherwise. That's a smaller, more boring rule than "think like an agent," and it's the one that actually changes what ships.