Building Agents with Observability: Metrics, Traces, Logs & Semantic Telemetry for LLM Workflows

Overview

Modern AI agents behave more like distributed systems than traditional apps: they plan, revise, call tools, update memory, recurse, hallucinate, retry, and occasionally wander into the void. Yet—most teams instrument them like chatbots.

The mismatch is the whole problem. A chatbot is a request-response system: one prompt in, one completion out, maybe a token counter for billing. An agent is a control loop that runs for many turns, accumulating state and making consequential decisions along the way. When a chatbot misbehaves you re-read the prompt. When an agent misbehaves on turn fourteen, the cause might be a stale memory write from turn three, a tool that silently returned an empty result on turn nine, and a planner that quietly doubled its branching factor in between. None of that is visible from the final answer alone.

If you can't see how your agent thinks, why it chose an action, what it saw last turn, or where its behavior drifted, then you can't reliably run it in production.

This post is a practical, engineering-oriented guide to observability for agentic systems, covering:

• What observability means in the context of AI agents
• Essential telemetry dimensions: metrics, traces, logs, events, semantic state
• Instrumentation patterns for:
  • persona drift detection
  • planning step explosion
  • hallucination frequency
  • tool bottleneck analysis
  • safety-trigger heatmaps
  • memory-inconsistency tracking
• Full copy-pasteable Python code for instrumentation wrappers
• How to map all of it onto the OpenTelemetry GenAI conventions when you outgrow the in-memory version

This is not conceptual fluff—you'll get the exact abstractions and collection pipelines you need for real agents.

What “Agent Observability” Actually Means

Observability for agents borrows from distributed systems, but extends it. The classic discipline assumes that if you can measure a system's outputs precisely enough, you can infer what it is doing internally. That assumption mostly holds for stateless services. It breaks down for agents, because an agent's most important internal state is semantic—what it currently believes, what role it thinks it is playing, what it thinks it is allowed to do—and none of that shows up in a latency histogram.

Traditional observability (3 pillars)

• Metrics — numerical indicators (latency, token usage, error counters)
• Logs — unstructured debugging streams (steps, actions, thoughts)
• Traces — structured spans connected across calls

Agent observability adds three agent-specific surfaces:

(1) Semantic Telemetry

Information about meaning, not just operations:

• What did the agent believe this step?
• Did its persona change?
• Did its memory contradict reality?
• Did it hallucinate a tool or permission?
• Did it misinterpret an observation?

Semantic telemetry is the part most teams skip, because it is the part that does not fall out of your HTTP layer for free. You have to generate it deliberately—usually with embeddings, a secondary checker model, or simple structural assertions about the agent's output. The rest of this post spends most of its code budget here, because this is where the production failures actually live.

(2) Cognitive Tracing

LLM “reasoning spans” and “action spans”—like distributed tracing but across the internal mental steps of an agent. A single user request fans out into a tree: a planning span, several LLM-call spans nested under it, tool-call spans nested under those, and memory-update spans threaded throughout. Reading that tree top to bottom is how you reconstruct “what the agent was thinking” after the fact, the same way you read a request trace to reconstruct a slow API call.

(3) Control-plane Health

Not just “did it run,” but:

• Is planning depth exploding?
• Is the agent stuck in loops?
• Are tools responding too slowly?
• Are safety/guardrails firing repeatedly?

This gives us six mission-critical observability objectives:

1. Persona drift monitoring
2. Tool-latency bottlenecks
3. Planning-step explosion detection
4. Hallucination tracking per step
5. Safety-trigger heatmaps
6. Memory consistency validation

We'll implement all six in code.

The Observability Architecture (Simple but Powerful)

A solid starting architecture:

+-------------------------+
|     Agent Orchestrator  |
|   (plan/act/reflect)    |
+-----------+-------------+
            |
            v
+-------------------------+
| Instrumentation Layer   |
|  - Metrics              |
|  - Logs                 |
|  - Traces               |
|  - Semantic Telemetry   |
+-----------+-------------+
            |
            v
+------------------+----------------------+
| Backends:                                   |
|  - Prometheus (metrics)                     |
|  - OpenTelemetry (traces & logs)            |
|  - S3 / SQL / BigQuery (step logs)          |
|  - Vector DB (semantic telemetry history)   |
+---------------------------------------------+

The key design decision is that the instrumentation layer is a single seam the orchestrator talks to, and the backends are swappable behind it. Your agent code never imports Prometheus or an OTEL exporter directly; it calls into one telemetry object, and that object fans out. This keeps the agent loop readable and lets you start with a dictionary in memory and graduate to a real collector without touching business logic.

For simplicity, we implement everything in pure Python but compatible with:

• Prometheus + Grafana
• OpenTelemetry (OTEL)
• Honeycomb / Datadog APM
• Elastic
• Any LLM provider

The Instrumentation Layer (Code Included)

We create a thin wrapper that sits around every LLM call, tool call, memory update, and plan step. Each of those four becomes a span, and the spans share one parent context so they reconstruct into a single trace per agent run. The in-memory version below is deliberately tiny—it is the mental model, not the production dependency. Later we show how the exact same shape maps onto OpenTelemetry spans.

Instrumentation Base Class

# observability.py
import time
import uuid
from dataclasses import dataclass, field
from typing import Dict, Any, List, Optional

@dataclass
class TraceSpan:
    span_id: str
    parent_id: Optional[str]
    name: str
    start_time: float
    end_time: Optional[float] = None
    attributes: Dict[str, Any] = field(default_factory=dict)

    def finish(self):
        self.end_time = time.time()

    @property
    def duration_ms(self) -> Optional[float]:
        if self.end_time is None:
            return None
        return (self.end_time - self.start_time) * 1000

@dataclass
class AgentTelemetry:
    traces: List[TraceSpan] = field(default_factory=list)
    logs: List[Dict[str, Any]] = field(default_factory=list)
    metrics: Dict[str, float] = field(default_factory=dict)

    def new_span(self, name: str, parent_id=None, **attrs):
        span = TraceSpan(
            span_id=str(uuid.uuid4()),
            parent_id=parent_id,
            name=name,
            start_time=time.time(),
            attributes=attrs,
        )
        self.traces.append(span)
        return span

    def log(self, event: str, **fields):
        self.logs.append({"event": event, **fields, "timestamp": time.time()})

    def incr(self, metric: str, amt: float = 1.0):
        self.metrics[metric] = self.metrics.get(metric, 0.0) + amt

This gives us:

• Structured traces
• Generic logs
• Numeric metrics

The one addition worth calling out is the duration_ms property. Computing durations once on the span, rather than re-deriving them in five different wrappers, keeps latency reporting consistent and means a span that never calls finish() shows up as unfinished instead of silently reporting zero. Now let's instrument each agent component.

Observability for LLM Calls (Latency, Token Counts, Semantic Evaluation)

Wrap LLM calls to collect:

• request/response
• token usage
• latency
• hallucination indicators
• persona drift signals
• safety-trigger flags

# llm_wrapper.py
from observability import AgentTelemetry
import time

def call_llm(obs: AgentTelemetry, role: str, prompt: str, system: str = None):
    span = obs.new_span("llm_call", role=role, prompt=prompt[:200])
    start = time.time()

    # TODO: wire actual provider, e.g.
    #   resp = client.chat.completions.create(model=..., messages=[...])
    #   response = resp.choices[0].message.content
    #   usage = resp.usage
    response = "<LLM output>"
    usage = {"input_tokens": 0, "output_tokens": 0}

    latency = time.time() - start
    span.attributes.update({
        "latency_ms": latency * 1000,
        "prompt_length": len(prompt),
        "response_length": len(response),
        "input_tokens": usage["input_tokens"],
        "output_tokens": usage["output_tokens"],
    })
    span.finish()
    obs.incr("llm.calls")
    obs.incr("llm.input_tokens", usage["input_tokens"])
    obs.incr("llm.output_tokens", usage["output_tokens"])

    # LOGGING
    obs.log("llm_call", role=role, latency_ms=latency * 1000)

    # SEMANTIC TELEMETRY STUBS
    # Replace these substring checks with real evals or a secondary checker model.
    if "I cannot" in response:
        obs.incr("llm.refusals")
    if "delete database" in response:
        obs.incr("llm.dangerous_suggestions")

    return response

Recording token usage as both span attributes and counters is worth the two extra lines: the span lets you see tokens per individual call when you are debugging one trace, and the counter lets you graph aggregate spend per run without re-parsing every span. This stub is also where you hook in:

• LLM guardrails
• Hallucination checkers
• Persona drift models
• Safety-classifier models

The substring checks shown here are intentionally crude—they exist to mark the spot, not to ship. In production the if "I cannot" in response line becomes a call to a real refusal classifier, and the dangerous-suggestion check becomes a proper safety model. Keeping the seam here means those upgrades never touch the agent loop.

Observability for Planning (Detecting Step Explosion)

Planning is where agents quietly burn money and time. A reasoning strategy that branches—Tree of Thoughts, Graph of Thoughts, or any planner that expands multiple candidate steps before committing—can multiply its own workload turn over turn. The failure mode is rarely a crash; it is an agent that keeps planning, never acts, and racks up a long tail of LLM calls. Counters make that pattern visible. Add counters for:

• steps per plan
• branching factor in ToT/GoT
• plan depth
• time per step

# planning_obs.py
from observability import AgentTelemetry

# Tune to your workload; this is a heuristic alarm, not a hard limit.
STEP_EXPLOSION_THRESHOLD = 50

def instrument_plan_step(obs: AgentTelemetry, step_num: int, content: str):
    obs.log("plan_step", step=step_num, content=content[:200])
    obs.incr("planning.steps")

    if obs.metrics["planning.steps"] > STEP_EXPLOSION_THRESHOLD:
        obs.incr("planning.explosions")

This catches runaway planners (common in ToT & GoT). The threshold of 50 is a starting heuristic, not a universal constant—pull it from a config so you can set it per task type, since a deep research agent legitimately plans far more than a simple form-filler. Once planning.explosions starts incrementing, that is your signal to either cap the loop or inspect why the planner refuses to commit to an action.

Observability for Tools (Latency, Failures, Overuse)

Tools are the agent's contact with the outside world, and they fail in all the ordinary ways external dependencies fail: timeouts, rate limits, malformed arguments, empty results that the agent then reasons about as if they were real. Wrapping every tool call in a span gives you per-tool latency and error rates, and—just as importantly—an exception that is caught here never crashes the agent loop, it becomes a recorded error the agent can react to.

# tool_wrapper.py
from observability import AgentTelemetry
import time

SLOW_TOOL_THRESHOLD_S = 2.0

def call_tool(obs: AgentTelemetry, tool_name: str, args: dict, func):
    span = obs.new_span("tool_call", name=tool_name, args=args)
    start = time.time()

    try:
        result = func(args)
        error = None
    except Exception as e:
        result = None
        error = str(e)
        obs.incr(f"tool.{tool_name}.errors")

    latency = time.time() - start
    span.attributes.update({"latency_ms": latency * 1000, "error": error})
    span.finish()

    obs.incr(f"tool.{tool_name}.calls")
    if latency > SLOW_TOOL_THRESHOLD_S:
        obs.incr(f"tool.{tool_name}.slow_calls")

    obs.log("tool_call", tool_name=tool_name, latency_ms=latency * 1000, error=error)
    return result, error

Recording error on the span as well as the counter is what lets you answer “which call failed and why,” not just “how many failed.” Key metrics you'll visualize in Grafana:

• per-tool latency
• failure counts
• slow-call heatmaps
• number of tool calls per run

Observability for Memory (Inconsistency & Drift)

Agents with memory (vector DB, key-value memory, episodic logs) can drift or contradict themselves. Memory bugs are the worst kind because they are invisible at the point of failure: the agent confidently asserts something it “remembers,” and the contradiction only surfaces several turns later when it acts on the stale value. Instrumenting writes gives you a paper trail of how each key evolved.

We track:

• contradictions
• inconsistent updates
• memory growth
• invalid keys

# memory_obs.py
from observability import AgentTelemetry

MEMORY_BLOAT_THRESHOLD = 2000  # characters per value; tune per store

def record_memory_update(obs: AgentTelemetry, key: str, old: str, new: str):
    obs.log("memory_update", key=key, new_value=new[:200])
    obs.incr("memory.writes")

    if old and old != new:
        obs.incr("memory.contradictions")

    if len(new) > MEMORY_BLOAT_THRESHOLD:
        obs.incr("memory.bloat")

Note that old != new only catches overwrites of the same key—it flags churn, not genuine semantic contradiction. A value can change without contradicting anything, and two consistent-looking values can still contradict in meaning. For the harder case, add a lightweight semantic check via an LLM: “Does the new memory contradict the old one?” and increment memory.contradictions only when it answers yes. That keeps cheap structural metrics and expensive semantic ones in the same counter namespace.

Observability for Persona Drift

Persona drift happens when an agent shifts:

• tone
• risk posture
• refusal style
• verbosity
• hedging
• alignment style

Drift matters because a long-running agent is repeatedly conditioned on its own prior output. Small stylistic shifts compound: an assistant that started cautious can become reckless over a long session, or a focused agent can slowly turn chatty and lose the plot. We track persona drift by:

1. Embedding the agent's prior responses
2. Embedding the current response
3. Computing cosine distance
4. Flagging deviations

# persona_drift.py
from observability import AgentTelemetry
import numpy as np

DRIFT_THRESHOLD = 0.25  # cosine distance; calibrate on your own traces

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def track_persona(obs: AgentTelemetry, emb_history: list, new_emb):
    if emb_history:
        drift = 1 - cosine(emb_history[-1], new_emb)
        obs.metrics["persona.drift"] = drift
        obs.log("persona_drift", value=drift)

        if drift > DRIFT_THRESHOLD:
            obs.incr("persona.drift_events")

    emb_history.append(new_emb)

Two practical notes. First, comparing only against the previous turn measures step-to-step jitter; comparing against a fixed reference embedding of the intended persona measures cumulative drift away from baseline. Most teams want both signals. Second, the 0.25 threshold is arbitrary until you calibrate it—run the detector over known-good sessions, look at the distribution of distances, and set the alarm above the normal noise floor rather than at a number copied from a blog post.

Observability for Hallucinations (Frequency, Severity, Type)

We can classify hallucinations by:

• nonexistent tool
• nonexistent permission
• invented fact
• invented memory
• invented intermediate plan step

The category matters as much as the count. An invented fact is a quality problem; a hallucinated tool or permission is a safety problem, because the agent is about to attempt something it has no business attempting. Tracking type lets you route the two differently. Use a secondary LLM for classification:

# hallucination_detector.py
from observability import AgentTelemetry
from llm_wrapper import call_llm

VALID_LABELS = {"none", "factual", "permission", "tool", "reasoning", "memory"}

def classify_hallucination(obs: AgentTelemetry, response: str):
    prompt = f"""
    Does the following response contain a hallucination?
    Classify as exactly one of: none | factual | permission | tool | reasoning | memory

    Response:
    {response}

    Output only the category, nothing else.
    """
    label = call_llm(obs, "hallucination_checker", prompt).strip().lower()
    if label not in VALID_LABELS:
        label = "unparseable"
    obs.incr(f"hallucination.{label}")
    return label

Validating the label against an allow-list matters more than it looks: the checker is itself an LLM, so it can return prose, punctuation, or an unexpected word, and an unvalidated f"hallucination.{label}" would silently create a junk metric name for every malformed reply. Folding everything off-list into a single unparseable bucket keeps your metric cardinality bounded and surfaces a flaky checker as its own signal. One caveat: the checker runs through the same call_llm wrapper, so its calls land in llm.calls too—either accept that or give checker traffic its own role prefix when you split dashboards.

Safety Trigger Heatmaps

Safety telemetry answers a question the other metrics cannot: not “did the agent do something unsafe” but “where in its run did the guardrails have to intervene.” Recording both the event type and the step number turns scattered incidents into a heatmap—you can see whether trouble clusters at the start of runs, at a particular tool, or only after the planner has been looping for a while.

# safety_obs.py
from observability import AgentTelemetry

def record_safety_event(obs: AgentTelemetry, event_type: str, step: int):
    obs.incr(f"safety.{event_type}")
    obs.log("safety_trigger", event=event_type, step=step)

Useful metrics:

• rate of overrides
• repeated unsafe intents
• unsafe tool invocations
• risky content frequency

A repeated unsafe intent is the one to watch most closely. A single guardrail trip is noise; the same unsafe intent firing three times in one run means the agent is trying to route around the guardrail rather than abandon the goal, and that is exactly the behavior you want paged about before it ships.

Putting It Together: Instrumented Agent Loop

Here's a full ReAct-style agent with observability integrated. ReAct interleaves a reasoning “thought” with an “action” and the resulting “observation,” looping until the agent finishes or hits a step cap—which makes it a natural fit for span-per-step instrumentation, since each loop iteration already corresponds to one unit of work.

# agent.py
from observability import AgentTelemetry
from llm_wrapper import call_llm
from tool_wrapper import call_tool
from planning_obs import instrument_plan_step
from hallucination_detector import classify_hallucination
from persona_drift import track_persona

MAX_STEPS = 20

def run_agent(task):
    obs = AgentTelemetry()
    embeddings = []
    history = []

    for step in range(MAX_STEPS):
        instrument_plan_step(obs, step, str(history))

        # LLM think step
        thought = call_llm(obs, "planner", f"Task: {task}\nHistory: {history}")
        classify_hallucination(obs, thought)

        # Extract action; no action means the agent is done
        if "Action:" not in thought:
            break
        action_line = thought.split("Action:", 1)[1].strip()
        tool_name, args = parse_action(action_line)

        # Tool execution
        result, error = call_tool(obs, tool_name, args, TOOLS[tool_name].func)
        history.append((thought, result))

        # Persona drift detection
        # embed() would come from a real model, e.g. sentence-transformers
        new_emb = fake_embed(thought)
        track_persona(obs, embeddings, new_emb)

    return obs

Every external dependency in this loop—the LLM, the tools, the checker—is reached through an instrumented wrapper, and every wrapper writes into the same obs object. That is the whole point of the architecture: the loop reads almost exactly like an uninstrumented ReAct agent, yet a single run produces:

• LLM traces
• Tool spans
• Safety events
• Persona drift signals
• Planning step metrics
• Hallucination counters
• Detailed logs

Everything can be pushed to:

• OTEL collector
• Prometheus
• Grafana
• Elasticsearch
• S3

Mapping to the OpenTelemetry GenAI Conventions

The in-memory AgentTelemetry object is the right way to learn the shape of the problem, but you should not invent your own attribute names for production. OpenTelemetry now defines GenAI semantic conventions—a standardized set of gen_ai.* attributes for model name, token usage, and operation type—developed by the GenAI special interest group. The conventions are still marked experimental and continue to evolve, but they are already recognized across major observability vendors, so emitting them now means your traces are portable instead of locked to one tool.

The translation is mechanical: replace your custom span with a real tracer span and rename the attributes onto the convention. The wrapper's public shape stays identical.

# llm_wrapper_otel.py
from opentelemetry import trace

tracer = trace.get_tracer("agent.observability")

def call_llm_otel(role: str, prompt: str, model: str = "gpt-4o"):
    with tracer.start_as_current_span("chat " + model) as span:
        # Standard GenAI convention attributes
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)

        # TODO: wire actual provider
        response = "<LLM output>"
        usage = {"input_tokens": 0, "output_tokens": 0}

        span.set_attribute("gen_ai.usage.input_tokens", usage["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", usage["output_tokens"])
        return response

A few names map directly: our input_tokens / output_tokens become gen_ai.usage.input_tokens / gen_ai.usage.output_tokens, and latency comes for free from span duration once a real backend records start and end times. The agent-specific surfaces from earlier—persona drift, hallucination type, safety triggers—do not yet have standard convention keys, so keep those under your own namespace as custom attributes. Use start_as_current_span with a with block so nested tool and checker spans automatically attach to the right parent and your reasoning tree reconstructs correctly.

Visualizing the Observability

Collected telemetry is only useful once someone can look at it. The three backends below each answer a different question—aggregate trends, a single run end to end, and semantic state over time—so most production setups run all three rather than picking one.

Grafana

• llm.calls → request volume
• tool.<name>.latency_ms → bottleneck heatmap
• hallucination.* → stacked bar chart
• persona.drift → line graph across time
• planning.steps → histogram to detect explosions

OTEL trace viewer

• Full step-by-step reasoning
• LLM spans
• Tool spans
• Memory update spans

Vector DB (semantic telemetry)

• persona drift embeddings
• memory consistency embeddings
• state comparisons

The division of labor is the takeaway: Grafana tells you something is wrong across the fleet, the trace viewer lets you walk one bad run thought by thought, and the vector store is where you go to ask the semantic questions—“how far has this agent drifted from its baseline persona this week”—that neither a counter nor a span can answer on its own.