Building Agents with Observability: Metrics, Traces, Logs & Semantic Telemetry for LLM Workflows

Overview

A chatbot is a request-response system: one prompt in, one completion out. When something goes wrong, you re-read the prompt and you're usually done. An agent doesn't work that way. It runs a control loop for many turns, calling tools, updating its own memory, revising a plan, and carrying state forward that nothing forces it to keep straight.

Take a support agent that triages refund requests over an eight-turn conversation: it reads the order, checks a return-window policy, calls a refund API, and writes a note back to the ticket. Say it refunds an order that was outside the window. The final action, the refund call, executed correctly. The mistake happened three turns earlier, when the agent misread the policy lookup and wrote "eligible" into its own working notes. Nothing in a request log shows that. The call that actually failed looks like a completely normal API call.

That's the shape of most agent incidents worth caring about. The wrong action is a symptom. The cause sits upstream, in a belief the agent formed and then acted on, and finding it means walking back through what the agent thought at each step, not just what it did.

What Each Signal Answers

Metrics, logs, and traces are still the foundation. They just don't cover everything an agent incident needs. Metrics tell you a call happened and how long it took. Traces tell you the order calls happened in, and logs fill in whatever unstructured detail neither one captures. None of the three tells you what the agent believed — which is the question that actually matters once you're staring at a wrong output instead of a slow one.

That's what semantic telemetry is for: signal about meaning rather than mechanics, generated on purpose, that answers whether the agent's tone shifted, whether it contradicted a stored fact, or whether it invented a tool that doesn't exist. It doesn't fall out of instrumenting your HTTP calls. You build it, usually with an embedding comparison, a secondary checker model, or a plain structural assertion on the output.

We'd start every agent's instrumentation plan with this table, before writing a line of collection code, because it forces you to name the failure before you build the sensor for it:

The table only helps if you can actually walk from "wrong output" to "the run that produced it" to "the belief that run was operating on." That's a diagnosis path, not a dashboard:

Figure 1 — Working backward from a bad output to the signal that explains it. The path only exists if the semantic layer was built before it was needed.

Instrumenting the Loop

The pattern underneath all of this is a thin wrapper around every LLM call, tool call, and memory update, writing into one shared object so a full run reconstructs as a single trace.

# observability.py
import time, uuid
from dataclasses import dataclass, field
from typing import Dict, Any, List, Optional

@dataclass
class TraceSpan:
    span_id: str
    parent_id: Optional[str]
    name: str
    start_time: float
    end_time: Optional[float] = None
    attributes: Dict[str, Any] = field(default_factory=dict)

    def finish(self):
        self.end_time = time.time()

    @property
    def duration_ms(self) -> Optional[float]:
        return None if self.end_time is None else (self.end_time - self.start_time) * 1000

@dataclass
class AgentTelemetry:
    traces: List[TraceSpan] = field(default_factory=list)
    logs: List[Dict[str, Any]] = field(default_factory=list)
    metrics: Dict[str, float] = field(default_factory=dict)

    def new_span(self, name: str, parent_id=None, **attrs):
        span = TraceSpan(str(uuid.uuid4()), parent_id, name, time.time(), attributes=attrs)
        self.traces.append(span)
        return span

    def log(self, event: str, **fields):
        self.logs.append({"event": event, **fields, "timestamp": time.time()})

    def incr(self, metric: str, amt: float = 1.0):
        self.metrics[metric] = self.metrics.get(metric, 0.0) + amt

A span that never calls finish() shows up as unfinished rather than silently reporting zero duration. That matters more than it looks: a hung tool call and a fast one should not report the same latency.

Wrap the LLM call itself the same way, and put the semantic checks in the same function so they can never be skipped by accident:

def call_llm(obs: AgentTelemetry, role: str, prompt: str):
    span = obs.new_span("llm_call", role=role, prompt=prompt[:200])
    start = time.time()

    response = provider_call(prompt)  # wire your actual client here

    span.attributes["latency_ms"] = (time.time() - start) * 1000
    span.finish()
    obs.incr("llm.calls")
    obs.log("llm_call", role=role, latency_ms=span.attributes["latency_ms"])

    # semantic telemetry stub — replace with a real classifier before shipping
    if "I cannot" in response:
        obs.incr("llm.refusals")
    return response

Wrap tool calls the same way, and catch the exception inside the wrapper rather than letting it crash the loop. A caught error becomes a recorded event the agent can react to; an uncaught one just ends the run.

def call_tool(obs: AgentTelemetry, name: str, args: dict, func):
    span = obs.new_span("tool_call", name=name, args=args)
    start = time.time()
    try:
        result, error = func(args), None
    except Exception as e:
        result, error = None, str(e)
        obs.incr(f"tool.{name}.errors")

    span.attributes.update({"latency_ms": (time.time() - start) * 1000, "error": error})
    span.finish()
    obs.incr(f"tool.{name}.calls")
    return result, error

That's the whole seam. Everything below is a different signal written into the same obs object through the same pattern.

Planning and Persona Drift

Two failure modes share a shape: neither one crashes anything, and both get worse gradually enough that nobody notices until the run is already expensive or the output is already wrong.

A planner that branches, Tree-of-Thoughts, Graph-of-Thoughts, anything that expands multiple candidate steps before committing, can multiply its own workload turn over turn. It rarely crashes. It just keeps planning, never quite commits to an action, and racks up a long tail of LLM calls that nobody bills to a bug until the invoice arrives.

STEP_EXPLOSION_THRESHOLD = 50  # heuristic, tune per workload

def instrument_plan_step(obs: AgentTelemetry, step_num: int):
    obs.incr("planning.steps")
    if obs.metrics["planning.steps"] > STEP_EXPLOSION_THRESHOLD:
        obs.incr("planning.explosions")

Persona drift is the same shape applied to tone instead of step count. A long-running agent is repeatedly conditioned on its own prior output, so small stylistic shifts compound: cautious can slide into reckless over a session with nobody changing a single instruction.

def track_persona(obs: AgentTelemetry, history: list, new_emb, threshold=0.25):
    if history:
        drift = 1 - cosine(history[-1], new_emb)
        obs.metrics["persona.drift"] = drift
        if drift > threshold:
            obs.incr("persona.drift_events")
    history.append(new_emb)

Comparing only against the previous turn catches step-to-step jitter. Comparing against a fixed reference embedding of the intended persona catches slow drift away from baseline instead. We'd track both. They answer different questions, and the threshold that's right for one is usually wrong for the other.

Memory and Hallucinations

Memory bugs and hallucinations fail the same way: invisibly, at the point of failure. The agent confidently asserts something it "remembers," or invents something outright, and the contradiction only shows up turns later, when it acts on the bad value.

def record_memory_update(obs: AgentTelemetry, key: str, old: str, new: str):
    obs.incr("memory.writes")
    if old and old != new:
        obs.incr("memory.contradictions")
    obs.log("memory_update", key=key)

old != new only flags an overwrite of the same key. It catches churn, not meaning: a value can change without contradicting anything, and two differently-worded values can still agree. Treat it as a cheap first pass, not the check.

Hallucinations are worth classifying by type, not just counting, because the type decides where the failure routes to. An invented fact is a quality problem. A hallucinated tool or permission is a safety problem, because the agent is about to attempt something it has no business attempting.

VALID_LABELS = {"none", "factual", "permission", "tool", "reasoning", "memory"}

def classify_hallucination(obs: AgentTelemetry, response: str, classifier_call):
    label = classifier_call(response).strip().lower()
    if label not in VALID_LABELS:
        label = "unparseable"
    obs.incr(f"hallucination.{label}")
    return label

Validating against an allow-list matters because the classifier is itself a model: an unvalidated label turns every malformed reply into its own junk metric name. Folding anything off-list into unparseable keeps the metric space bounded, and turns a flaky classifier into its own visible signal instead of noise scattered across a hundred one-off counters.

Safety Triggers

Safety telemetry answers a narrower question than the rest: not whether the agent did something unsafe, but where in the run the guardrails had to step in.

def record_safety_event(obs: AgentTelemetry, event_type: str, step: int):
    obs.incr(f"safety.{event_type}")
    obs.log("safety_trigger", event=event_type, step=step)

Recording the step number alongside the event type turns scattered incidents into something you can actually read: whether trouble clusters early in runs, around one specific tool, or only after the planner has been looping for a while.

OpenTelemetry Mapping

The in-memory AgentTelemetry object above is the right way to learn the shape of the problem. It is not what you should run in production, because inventing your own attribute names locks every trace to whatever tool reads that schema today.

OpenTelemetry's GenAI semantic conventions define standard gen_ai.* attributes for model name, token usage, and operation type. They're still marked experimental, but emitting them now buys you a shot at portability as the convention matures, instead of locking every trace to a schema you invented yourself.

from opentelemetry import trace

tracer = trace.get_tracer("agent.observability")

def call_llm_otel(model: str, prompt: str):
    with tracer.start_as_current_span("chat " + model) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", model)
        response = provider_call(prompt)
        span.set_attribute("gen_ai.usage.input_tokens", 0)  # wire real usage
        return response

The translation is mechanical for the parts OpenTelemetry already models: token counts and latency map straight across. The agent-specific surfaces, persona drift, hallucination type, safety triggers, don't have standard keys yet. Keep those as custom attributes under your own namespace until the convention catches up, and use start_as_current_span so nested tool and checker spans attach to the right parent automatically.

Conclusion

None of the signals above replace each other. Metrics and traces tell you the fleet is healthy and show you what ran, in what order. Semantic telemetry is the only one of the four that tells you what the agent believed, and it's the one that answers the question you actually have at 2am: not "is it up," but "why did it do that."

We think most teams don't actually skip observability. They build the half that looks like ordinary distributed tracing and stop there, because that half is the one that falls out of the HTTP layer for free. The half that catches a hallucinated permission or a drifted persona has to be built on purpose, before the run that needs it.