Building Agents with Observability: Metrics, Traces, Logs & Semantic Telemetry for LLM Workflows

Overview

A chatbot is a request-response system: one prompt in, one completion out. When something goes wrong, you re-read the prompt and you're usually done. An agent doesn't work that way. It runs a control loop for many turns, calling tools, updating its own memory, revising a plan, and carrying state forward that nothing forces it to keep straight.

Take a support agent that triages refund requests over an eight-turn conversation: it reads the order, checks a return-window policy, calls a refund API, and writes a note back to the ticket. Say it refunds an order that was outside the window. The final action, the refund call, executed correctly. The mistake happened three turns earlier, when the agent misread the policy lookup and wrote "eligible" into its own working notes. Nothing in a request log shows that. The call that actually failed looks like a completely normal API call.

That's the shape of most agent incidents worth caring about, and the research that has tried to catalogue them agrees. A 2025 study of multi-agent failures built a taxonomy from 150 execution traces and landed on "14 unique modes, clustered into 3 categories: (i) system design issues, (ii) inter-agent misalignment, and (iii) task verification", validated across 1,600-plus annotated traces from seven frameworks with an inter-annotator agreement of κ = 0.88. Almost none of those fourteen modes is a crash.

What Each Signal Answers

Metrics, logs, and traces are still the foundation. Metrics tell you a call happened and how long it took. Traces tell you the order calls happened in, and logs fill in whatever unstructured detail neither one captures. None of the three tells you what the agent believed, which is the question you actually have once you're staring at a wrong output instead of a slow one.

That's what semantic telemetry is for: signal about meaning rather than mechanics, generated on purpose, that answers whether the agent's tone shifted, whether it contradicted a stored fact, or whether it invented a tool that doesn't exist. It doesn't fall out of instrumenting your HTTP calls. You build it, usually with an embedding comparison, a secondary checker model, or a plain structural assertion on the output.

We'd start every agent's instrumentation plan with this table, before writing a line of collection code, because it forces you to name the failure before you build the sensor for it:

Table 1 — Seven questions, and which layer can answer each. The last row is the one a normal service never has to ask.

The table only helps if you can actually walk from "wrong output" to "the run that produced it" to "the belief that run was operating on." That's a diagnosis path, not a dashboard:

Figure 1 — Working backward from a bad output to the signal that explains it. The path only exists if the semantic layer was built before it was needed.

Instrumenting the Loop

The pattern underneath all of this is a thin wrapper around every LLM call, tool call, and memory update, writing into one shared object so a full run reconstructs as a single trace.

# observability.py
import time, uuid
from dataclasses import dataclass, field
from typing import Dict, Any, List, Optional

@dataclass
class TraceSpan:
    span_id: str
    parent_id: Optional[str]
    name: str
    start_time: float
    end_time: Optional[float] = None
    attributes: Dict[str, Any] = field(default_factory=dict)

    def finish(self):
        self.end_time = time.time()

    @property
    def duration_ms(self) -> Optional[float]:
        return None if self.end_time is None else (self.end_time - self.start_time) * 1000

@dataclass
class AgentTelemetry:
    traces: List[TraceSpan] = field(default_factory=list)
    logs: List[Dict[str, Any]] = field(default_factory=list)
    metrics: Dict[str, float] = field(default_factory=dict)

    def new_span(self, name: str, parent_id=None, **attrs):
        span = TraceSpan(str(uuid.uuid4()), parent_id, name, time.time(), attributes=attrs)
        self.traces.append(span)
        return span

    def log(self, event: str, **fields):
        self.logs.append({"event": event, **fields, "timestamp": time.time()})

    def incr(self, metric: str, amt: float = 1.0):
        self.metrics[metric] = self.metrics.get(metric, 0.0) + amt

A span that never calls finish() shows up as unfinished rather than silently reporting zero duration. That matters more than it looks: a hung tool call and a fast one should not report the same latency.

Wrap the LLM call itself the same way, and put the semantic checks in the same function so they can never be skipped by accident:

def call_llm(obs: AgentTelemetry, role: str, prompt: str):
    span = obs.new_span("llm_call", role=role, prompt=prompt[:200])
    start = time.time()

    response = provider_call(prompt)  # wire your actual client here

    span.attributes["latency_ms"] = (time.time() - start) * 1000
    span.finish()
    obs.incr("llm.calls")
    obs.log("llm_call", role=role, latency_ms=span.attributes["latency_ms"])

    # semantic telemetry stub — replace with a real classifier before shipping
    if "I cannot" in response:
        obs.incr("llm.refusals")
    return response

Wrap tool calls the same way, and catch the exception inside the wrapper rather than letting it crash the loop. A caught error becomes a recorded event the agent can react to; an uncaught one just ends the run.

def call_tool(obs: AgentTelemetry, name: str, args: dict, func):
    span = obs.new_span("tool_call", name=name, args=args)
    start = time.time()
    try:
        result, error = func(args), None
    except Exception as e:
        result, error = None, str(e)
        obs.incr(f"tool.{name}.errors")

    span.attributes.update({"latency_ms": (time.time() - start) * 1000, "error": error})
    span.finish()
    obs.incr(f"tool.{name}.calls")
    return result, error

That's the whole seam. Everything below is a different signal written into the same obs object through the same pattern.

One thing to decide before you ship it: whether the prompt[:200] on that span is allowed to leave the process. Prompt and completion text is the most useful field in an agent trace and the most dangerous one, which is why Azure's own instrumentation makes it opt-in behind an environment variable, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, rather than capturing it by default. Copy that default. A trace store that accumulates raw customer conversations for a 30-day retention window is a data-protection question, not an observability one.

Planning and Persona Drift

Two failure modes share a shape: neither one crashes anything, and both get worse gradually enough that nobody notices until the run is already expensive or the output is already wrong.

A planner that branches, Tree-of-Thoughts, Graph-of-Thoughts, anything that expands multiple candidate steps before committing, can multiply its own workload turn over turn. It rarely crashes. It just keeps planning, never quite commits to an action, and racks up a long tail of LLM calls that nobody bills to a bug until the invoice arrives.

You do not have to invent the alert threshold. OpenTelemetry's GenAI conventions already define gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls as histograms over one agent invocation, with bucket boundaries running [1, 2, 4, ... 128]. The top bucket is the convention's own statement about what "a lot" looks like for a single agent run, and it is a better starting point than a number you made up.

STEP_EXPLOSION_THRESHOLD = 128  # top bucket of gen_ai.invoke_agent.tool_calls

def instrument_plan_step(obs: AgentTelemetry, step_num: int):
    obs.incr("planning.steps")
    if obs.metrics["planning.steps"] > STEP_EXPLOSION_THRESHOLD:
        obs.incr("planning.explosions")

Persona drift is the same shape applied to tone instead of step count, and it has a measured onset. A study of instructed self-chats between LLaMA2-chat-70B and GPT-3.5 found "a significant instruction drift within eight rounds of conversations", and attributed it to "attention decay over long exchanges" in the transformer attention mechanism. Eight turns is not a long session. It is one support conversation.

def track_persona(obs: AgentTelemetry, history: list, new_emb, threshold=0.25):
    if history:
        drift = 1 - cosine(history[-1], new_emb)
        obs.metrics["persona.drift"] = drift
        if drift > threshold:
            obs.incr("persona.drift_events")
    history.append(new_emb)

Comparing only against the previous turn catches step-to-step jitter. Comparing against a fixed reference embedding of the intended persona catches slow drift away from baseline instead. We'd track both, and we'd put the turn index on every drift sample, because if the eight-round result generalises at all then the interesting question is not whether drift happens but where in the conversation it starts.

Memory and Hallucinations

Memory bugs and hallucinations fail the same way: invisibly, at the point of failure. The agent confidently asserts something it "remembers," or invents something outright, and the contradiction only shows up turns later, when it acts on the bad value.

def record_memory_update(obs: AgentTelemetry, key: str, old: str, new: str):
    obs.incr("memory.writes")
    if old and old != new:
        obs.incr("memory.contradictions")
    obs.log("memory_update", key=key)

old != new only flags an overwrite of the same key. It catches churn, not meaning: a value can change without contradicting anything, and two differently-worded values can still agree. Treat it as a cheap first pass, not the check.

Hallucinations are worth classifying by type, not just counting, because the type decides where the failure routes to. NIST's generative-AI profile calls the underlying behaviour confabulation, "the production of confidently stated but erroneous or false content", and lists it as one of twelve risks a measurement program is expected to cover. An invented fact is a quality problem. A hallucinated tool or permission is a safety problem, because the agent is about to attempt something it has no business attempting.

VALID_LABELS = {"none", "factual", "permission", "tool", "reasoning", "memory"}

def classify_hallucination(obs: AgentTelemetry, response: str, classifier_call):
    label = classifier_call(response).strip().lower()
    if label not in VALID_LABELS:
        label = "unparseable"
    obs.incr(f"hallucination.{label}")
    return label

Validating against an allow-list matters because the classifier is itself a model: an unvalidated label turns every malformed reply into its own junk metric name. Folding anything off-list into unparseable keeps the metric space bounded, and turns a flaky classifier into its own visible signal instead of noise scattered across a hundred one-off counters.

Safety Triggers

Safety telemetry answers a narrower question than the rest: not whether the agent did something unsafe, but where in the run the guardrails had to step in.

def record_safety_event(obs: AgentTelemetry, event_type: str, step: int):
    obs.incr(f"safety.{event_type}")
    obs.log("safety_trigger", event=event_type, step=step)

Recording the step number alongside the event type turns scattered incidents into something you can read: whether trouble clusters early in runs, around one specific tool, or only after the planner has been looping for a while. OWASP's LLM06:2025 Excessive Agency puts the same two controls at the end of its mitigation list, monitoring extension activity and rate-limiting, on the grounds that when the permissions themselves are too broad, the remaining defence is noticing fast.

The gen_ai Conventions, As They Actually Stand

The in-memory AgentTelemetry object above is the right way to learn the shape of the problem. It is not what you should run in production, because inventing your own attribute names locks every trace to whatever tool reads that schema today.

OpenTelemetry's GenAI semantic conventions are the portable option, with one caveat worth knowing before you bet on them: they moved. The gen_ai.* entries in the main semantic-conventions registry now carry a Deprecated marker and the note "Moved to the OpenTelemetry GenAI semantic conventions repository", and the conventions in that new repository are marked Development, meaning they remain subject to change. Emit them anyway. A convention in flux still beats a private schema.

The parts that matter for an agent:

Table 2 — What the GenAI conventions already model. Everything in the left column has a name you should use rather than invent.

Two things stand out. The convention has first-class operation names for agent invocation and for memory reads and writes, which means two of the three failure modes above already have a standard place to live. And the bucket ceilings tell you what the authors expect: a single agent invocation is allowed to take nearly seven minutes, and a single tool call is not.

Portability is still aspirational rather than real. The sample span in Azure's own tracing documentation carries gen_ai.system": "openai" alongside gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model and gen_ai.usage.input_tokens, while the current convention's required provider attribute is gen_ai.provider.name. Expect to normalise. Expect to normalise again.

from opentelemetry import trace

tracer = trace.get_tracer("agent.observability")

def call_llm_otel(model: str, provider: str, prompt: str):
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", provider)
        span.set_attribute("gen_ai.request.model", model)
        response = provider_call(prompt)
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
        # no standard key for this one yet
        span.set_attribute("agent.persona.drift", 0.0)
        return response

Persona drift, hallucination type and safety triggers have no standard keys. Keep them under your own namespace until the convention catches up, and use start_as_current_span so nested tool and checker spans attach to the right parent automatically.

Across process boundaries the parenting is the W3C Trace Context header doing the work. traceparent is four dash-separated fields, version plus a 32-character trace-id, a 16-character parent-id and 8 bits of flags, as in 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. It became a W3C Recommendation on 23 November 2021, which makes it the most stable thing in this entire stack. If a tool call leaves your process, propagate it, or the span tree ends at the boundary.

Figure 2 — The refund incident as a span tree. Every span here has a standard name except the last one, and the dotted line is the relationship you have to record yourself.

Retention and Audit

Agent telemetry and an agent audit trail are different artifacts with different lifetimes, and conflating them is how teams end up with a 14-day trace retention and a regulator asking about March.

Telemetry is sampled, lossy and tuned for debugging. An audit record is per-action, complete, and tuned for answering who did what on whose behalf: aact:audit-granularity/per-action-with-principal-chain rather than the aact:audit-granularity/session-level record sampling leaves you with. The EU AI Act asks for the second. Article 12 requires that high-risk systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system", explicitly so that risk situations can be identified and deployers can monitor operation. Article 19 then sets the retention floor for providers: logs kept "for a period appropriate to the intended purpose" of the system, and in any case "at least six months".

Six months of sampled traces is not the same asset as six months of action records, and it is usually far more expensive. So decide early which spans are also audit events. In the refund example, the execute_tool issue_refund span is an audit event and the four chat spans around it are not, even though the chat spans are what you'd actually want during an incident.

Where Fabriq Fits

The one piece of this that is awkward to build inside the agent is identity. A trace tells you an agent called a tool. It does not, by default, tell you which human's authority the call was made under, and that is the field an audit question turns on. Whose authority an agent acts with is the taxonomy's delegation model dimension.

Agentic Fabriq carries two identities on every request, the agent and the acting user, and writes both into an audit record alongside the call, with credentials held in a vault and attached server-side so the agent never handles the secret. That gives the execute_tool span an actor and a subject rather than just a tool name. Everything else here, the drift metric, the hallucination classifier, the safety counters, is still yours to build. There is no standard key for belief.

Frequently Asked Questions

What is semantic telemetry? Signal about what a model meant or believed, rather than about whether a call succeeded. Persona-drift distance, hallucination type, memory-contradiction flags and refusal counts are all semantic telemetry. It is produced deliberately, usually by an embedding comparison, a checker model or an assertion on the output, and none of it falls out of HTTP instrumentation.

Do the OpenTelemetry GenAI conventions cover agents? Partly. invoke_agent and create_agent are defined operation names, and there are agent-level metrics for duration, inference calls and tool calls. The conventions are marked Development and moved out of the main semconv repository into a dedicated GenAI repository, so pin what you emit and expect churn.

Which gen_ai attributes are required? On a GenAI span, gen_ai.operation.name and gen_ai.provider.name. Others, including gen_ai.request.model, are conditionally required or recommended depending on span type.

Should I log prompts and completions? Only deliberately. Azure's instrumentation gates message-content capture behind OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, and that is the right default for anyone whose prompts contain customer data. Capture the classification, the token counts and the decision; capture the text on a flag, into a store with its own retention rule.

How do I detect persona drift? Embed each agent turn and compare it both to the previous turn and to a fixed reference embedding of the intended persona. The first catches jitter, the second catches slow slide. Record the turn index with every sample, since drift has been measured setting in "within eight rounds of conversations".

How do I tell a hallucination from a tool failure? By where the trace stops. If an expected span is missing, timed out or returned empty, it is a tool failure. If every span ran and the output is still wrong, the failure is upstream in the model's belief, which is exactly the case no span tree explains on its own.

Why does the agent pass in testing and fail in production? Often because the test measured one attempt. τ-bench's pass^k metric exists for this: the same agents that "succeed on <50% of the tasks" single-shot drop to "pass^8 <25% in retail" when required to get all eight attempts right. If your eval reports a single number per task, it is not measuring the thing that breaks.

How long do I keep agent traces? Long enough for debugging, which is usually weeks, and separately long enough for audit, which under the EU AI Act's Article 19 is "at least six months" for high-risk systems and longer wherever sectoral law says so. Those are two retention policies, not one.

Conclusion

None of the signals above replace each other. Metrics and traces tell you the fleet is healthy and show what ran, in what order. Semantic telemetry is the only one that tells you what the agent believed, and it's the one that answers the question you actually have at 2am: not "is it up," but "why did it do that."

We think most teams don't skip observability. They build the half that looks like ordinary distributed tracing and stop there, because that half falls out of the HTTP layer for free. The half that catches a hallucinated permission or a drifted persona has to be built on purpose, before the run that needs it.

Emit the standard names where they exist. There are more of them than there were a year ago, and invoke_agent, execute_tool and the memory operations cover most of the mechanical surface. Then accept that the belief layer is yours, and name each signal after the failure it is there to catch.

Sources

All URLs read 2026-09-29.

• OpenTelemetry GenAI semantic conventions: spans — span naming, the required gen_ai.operation.name and gen_ai.provider.name, the operation-name enum, and Development status.
• OpenTelemetry GenAI semantic conventions: metrics — gen_ai.invoke_agent.duration, .inference_calls, .tool_calls, gen_ai.execute_tool.duration and their bucket boundaries.
• OpenTelemetry semantic conventions registry, gen_ai attributes — the Deprecated markers and the "Moved to the OpenTelemetry GenAI semantic conventions repository" note.
• W3C Trace Context — traceparent format, field sizes, and the 23 November 2021 Recommendation status.
• View trace results for AI applications, Microsoft Foundry — a real emitted span with gen_ai.* attributes, and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT as an opt-in.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — the pass^k metric, "<50% of the tasks", and "pass^8 <25% in retail".
• Why Do Multi-Agent LLM Systems Fail? — the 14-mode, 3-category failure taxonomy, 150 traces, 1,600-plus annotations, κ = 0.88.
• Measuring and Controlling Instruction (In)Stability in Language Model Dialogs — instruction drift "within eight rounds of conversations" and attention decay as the proposed mechanism.
• NIST AI 600-1, Generative AI Profile — confabulation defined as "confidently stated but erroneous or false content", within the twelve-risk list.
• Agent Access Control Taxonomy — the audit granularity and delegation model dimensions; per-action-with-principal-chain against session-level for what sampling costs you, and the delegation question a trace alone cannot answer.
• OWASP LLM06:2025 Excessive Agency — excessive functionality, permissions and autonomy, and monitoring plus rate-limiting as the fallback controls.
• EU AI Act Article 12, Record-keeping — automatic logging "over the lifetime of the system".
• EU AI Act Article 19, Automatically generated logs — the "at least six months" retention floor.