Integrating Confidence Scores Into Real Agent Frameworks (LangChain, LangGraph, AutoGen, Instructor)

Overview

Nobody wires confidence scoring into a blank Python file. It gets added three months in, to an agent that already lives inside LangChain, LangGraph, a multi-agent runtime, or Instructor, and the question is never "how do I build an agent." It is "how do I make the agent I already have admit when it is not sure," which is a plumbing question with a calibration question hiding behind it.

Each of these libraries encodes a different assumption about how control flow moves. LangChain composes a model call, tools and middleware into an agent loop. LangGraph is a state machine with named nodes and explicit edges. AG2 and the AutoGen successors are conversations between agents that pass messages. Instructor does not touch control flow at all; it guarantees the shape of what comes back. A score that plugs cleanly into one usually has to be rebuilt, not just re-typed, to plug into another.

Versions in this article were checked on 29 September 2026 against each project's own documentation and its PyPI release record: langchain 1.4.3, langchain-core 1.6.6, langgraph 1.2.12, ag2 1.1.1, autogen-agentchat 0.7.5, agent-framework 1.19.0, instructor 1.17.0. Anything below is worth re-checking against your own lockfile before you copy it, and the section on failure modes exists largely because API drift and bad calibration produce symptoms that look alike.

What the Number Actually Is

Almost every production confidence score is the model saying a number about itself. That is worth stating plainly, because the frameworks below will validate the number, route on it, and put it in a dashboard without ever touching whether it means anything.

The research here is more encouraging than instinct suggests, and more limited. Kadavath et al. found that "larger models are well-calibrated on diverse multiple choice and true/false questions" when the question is formatted appropriately, but that those same models "struggle with calibration of P(IK) on new tasks." Tian et al. compared asking the model for a number against reading the model's own token probabilities, and found the verbalized number better calibrated for RLHF-tuned models, "often reducing the expected calibration error by a relative 50%." Xiong et al., testing the same idea across more models, report that models "tend to be overconfident, potentially imitating human patterns of expressing confidence," and that no elicitation technique they tried consistently beat the others.

Put those three together and you get a usable working rule. Ask for the number rather than deriving it from logprobs. Expect it to skew high. Do not treat 0.9 as a probability until you have checked it against outcomes, which is what the calibration section is for. The expected calibration error you will be computing is the same quantity Guo et al. used in 2017 to show that "modern neural networks, unlike those from a decade ago, are poorly calibrated" — the diagnosis predates transformers.

LangChain: A Field in the Response Schema

Two things changed in LangChain 1.x and both matter here. Structured output on a model is still with_structured_output, undeprecated and documented. Structured output on an agent is now a response_format argument to create_agent, and it takes a strategy object rather than a bare schema.

Start with the schema. The ge/le constraints are Pydantic's, and they reject an out-of-range score before your code sees it.

from pydantic import BaseModel, Field

class AnswerWithConfidence(BaseModel):
    answer: str = Field(..., description="The answer to the question.")
    confidence: float = Field(
        ..., ge=0, le=1,
        description="How confident you are in the answer, from 0 to 1.",
    )

On a plain model call, bind it and invoke:

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-5.5", temperature=0)
structured = model.with_structured_output(AnswerWithConfidence, include_raw=True)

out = structured.invoke("What is the capital of Peru? Rate your confidence 0-1.")
result = out["parsed"]          # AnswerWithConfidence, or None
raw = out["raw"]                # AIMessage, carries usage_metadata

include_raw=True is the flag worth knowing. Without it a parse failure raises; with it you get a dict of raw, parsed and parsing_error, and the raw AIMessage carries usage_metadata with input_tokens, output_tokens and input_token_details for cache reads. That is how you attribute the cost of the confidence field itself.

Inside an agent, the choice is between two strategies, and it is a real decision rather than a style preference:

from langchain.agents import create_agent
from langchain.agents.structured_output import ToolStrategy, ProviderStrategy

# Provider-native structured output. Needs langchain>=1.1 for model profiles,
# and strict= needs langchain>=1.2.
agent = create_agent(
    model="openai:gpt-5.5",
    tools=[...],
    response_format=ProviderStrategy(AnswerWithConfidence, strict=True),
)

# Emulated through a tool call. Supports union types, and can repair itself.
agent = create_agent(
    model="openai:gpt-5.5",
    tools=[...],
    response_format=ToolStrategy(AnswerWithConfidence, handle_errors=True),
)

The parsed object lands in the "structured_response" key of the agent's result. ToolStrategy's handle_errors is the interesting parameter: set to True, a validation failure sends the model "error feedback" and retries; set to False, "exceptions propagate." It also accepts a string, an exception type, or a callable, so you can repair one class of failure and fail loudly on the rest.

One more thing shipped in 1.x that is easy to miss. RubricMiddleware applies "LLM-as-a-judge grading for self-evaluation and iteration," which is the critic pattern people used to hand-roll across two agents, available as a list entry on create_agent. If your reason for reaching for a multi-agent runtime was cross-model grading, check that middleware first.

LangGraph: Confidence as an Edge Condition

LangGraph makes each outcome a distinct node, so "confident," "needs clarification" and "fallback" each get their own logic, retries and downstream edges. Confidence goes into state, a router reads state and returns a label, and add_conditional_edges maps labels to nodes.

from typing import TypedDict, Optional

class AgentState(TypedDict):
    question: str
    answer: Optional[str]
    confidence: Optional[float]
    attempts: int

def answer_node(state: AgentState):
    res = structured.invoke(state["question"])["parsed"]
    return {"answer": res.answer, "confidence": res.confidence,
            "attempts": state["attempts"] + 1}

def router(state: AgentState) -> str:
    conf = state["confidence"] or 0.0
    if state["attempts"] >= 3:
        return "fallback"
    if conf > 0.8:
        return "confident"
    return "needs_clarification" if conf > 0.5 else "retry"

The wiring mistake that survives longest is passing a condition= argument to add_edge. There is no such argument. Branching is add_conditional_edges(source, router, path_map).

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(AgentState)
workflow.add_node("answer_node", answer_node)
workflow.add_node("clarify", lambda s: {"answer": "Can you clarify?"})
workflow.add_node("fallback", lambda s: {"answer": "Not confident enough."})

workflow.add_edge(START, "answer_node")
workflow.add_conditional_edges(
    "answer_node",
    router,
    {"confident": END, "needs_clarification": "clarify",
     "retry": "answer_node", "fallback": "fallback"},
)
workflow.add_edge("clarify", END)
workflow.add_edge("fallback", END)

graph = workflow.compile()

add_edge(START, ...) is the current idiom. set_entry_point still works and the docs call it equivalent, while naming add_edge(START, ...) and add_edge(..., END) "the recommended modern syntax." A router can also be replaced by returning Command(update=..., goto=...) from the node itself, which collapses the update and the branch into one step.

Figure 1 — The router is the only place the threshold appears. Every branch is a node you can give its own retry, timeout and log line.

AutoGen, AG2, and What Replaced Them

This is the section that aged worst, so here is the current state of it. Microsoft's autogen Python packages are at 0.7.5, released 30 September 2025, and the repository README carries a maintenance-mode banner: AutoGen "will not receive new features or enhancements and is community managed going forward," with new users pointed at Microsoft Agent Framework. The community AG2 fork, which inherited the pyautogen import path, is at 1.1.1 and released four times in September 2026 alone.

So the import in older tutorials now resolves to one of three different libraries depending on what you installed:

# AG2 1.x — pip install "ag2[openai]"
from autogen import ConversableAgent, LLMConfig

# AutoGen 0.7.x, maintenance mode — pip install autogen-agentchat autogen-ext[openai]
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

# Microsoft Agent Framework 1.x — pip install agent-framework
from agent_framework import Agent

The cross-checking pattern is the same in all three: one agent answers with a score, a second and cheaper agent grades the answer, and you keep both numbers. On AutoGen 0.7.x the surface is async and the model client is per agent.

import asyncio, json
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

async def scored_answer(question: str) -> dict:
    answerer = AssistantAgent(
        "answerer",
        model_client=OpenAIChatCompletionClient(model="gpt-5.5"),
        system_message='Reply with JSON only: {"answer": "...", "confidence": 0.0}',
    )
    critic = AssistantAgent(
        "critic",
        model_client=OpenAIChatCompletionClient(model="gpt-5-nano"),
        system_message="Rate the answer's factual accuracy. Reply with one number, 0-1.",
    )
    said = json.loads((await answerer.run(task=question)).messages[-1].content)
    graded = (await critic.run(task=f"Q: {question}\nA: {said['answer']}")).messages[-1].content
    return {"self": said["confidence"], "critic": float(graded.strip()),
            "answer": said["answer"]}

Note what the migration guide flags, because it changes the cost of this pattern: AssistantAgent "is single-turn unless you increase max_tool_iterations," while Agent Framework's Agent "is multi-turn by default and keeps invoking tools until it can return a final answer." Port the critic naively and you may buy a tool loop you did not budget for.

The gap between the two numbers is the part worth keeping. Agreement tells you little. A self-report of 0.95 against a critic score of 0.3 is a specific, loggable signal that the answering model is overconfident on this class of question, and it is more actionable than either number alone.

Instructor: The Field Is Always There

Instructor's job is narrow: the object that reaches your code satisfies your schema, or an exception does. Its documented constructor is now from_provider, which takes a provider-prefixed model string rather than a pre-built client.

import instructor
from pydantic import BaseModel, Field

class Response(BaseModel):
    answer: str
    confidence: float = Field(..., ge=0, le=1)

client = instructor.from_provider("openai/gpt-5.5")

result = client.chat.completions.create(
    response_model=Response,
    max_retries=3,
    messages=[{"role": "user", "content": "Explain the Doppler effect. Rate confidence 0-1."}],
)

Two behaviours are worth knowing before you set max_retries and forget it. Failed attempts "are automatically propagated to reask handlers," so the model sees its own validation error on the next try rather than guessing again blind. And when the retries run out you get InstructorRetryException, carrying n_attempts and a failed_attempts list where each entry has an attempt_number and the exception that ended it. That list is the single most useful artefact in this whole article for diagnosing a schema the model cannot satisfy, and most teams throw it away in a bare except.

from instructor.exceptions import InstructorRetryException

try:
    result = client.chat.completions.create(...)
except InstructorRetryException as e:
    log.warning("schema unsatisfied after %s attempts", e.n_attempts,
                extra={"failures": [f.exception for f in e.failed_attempts]})

max_retries also accepts a Tenacity Retrying or AsyncRetrying instance, which is how you get backoff and stop conditions instead of a bare integer.

Choosing the Wiring

The decision is not which library is better. It is which guarantee you need and what you are willing to pay for it.

Two practical readings of that table. If you only want the field to exist so a metrics pipeline can ingest it, Instructor is the whole answer and adding a graph is overhead. If you want the low-confidence path to be a real subflow with its own retries and timeouts, LangGraph is worth the state object, and you will still want a schema library inside the node.

Failure Modes

Four things go wrong, and two of them look identical from the outside.

The parse fails and the score defaults to zero. A stray code fence around the JSON, a trailing comment, a refusal. Symptom: a sudden cluster of low-confidence routes with no change in answer quality. Tell it apart from genuine low confidence by logging the raw text alongside the parsed object; a parse failure has no number in it at all, while real low confidence does.

The model is confidently wrong. Symptom: the top confidence bucket has materially worse accuracy than the bucket below it. This is not a bug you can fix in the wiring. It is the overconfidence the elicitation literature describes, and the fix is a higher threshold or a second grader.

The threshold drifts out from under you. You tuned 0.8 against one model, then changed models. Score distributions are not portable across models, and nothing in any of these libraries will warn you. Re-derive thresholds from the bucketed accuracy table on every model change, and store the model name next to every logged score so you can.

The retry loop does not terminate. Symptom: latency and spend climb while the answer never ships. This got quieter and more expensive at once when LangGraph's default recursion limit moved to 1,000 steps. An attempts counter in state is the fix; a recursion limit is the backstop, not the plan.

Logging and Calibration

Routing on confidence is half the job. The other half is finding out whether a high score is actually more likely to be correct, which needs the score logged next to the outcome.

If you are emitting OpenTelemetry, use the GenAI semantic conventions for the surrounding call rather than inventing attribute names: gen_ai.operation.name and gen_ai.provider.name are Required, gen_ai.request.model is Conditionally Required, and gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens are Recommended. Every one of those attributes is still marked Development stability, so pin your instrumentation version. The confidence score itself has no convention, so namespace it yourself and be consistent.

from opentelemetry import trace

tracer = trace.get_tracer("agent.confidence")

def answer_with_telemetry(question: str):
    with tracer.start_as_current_span("chat") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", "gpt-5.5")
        out = structured.invoke(question)
        res, raw = out["parsed"], out["raw"]
        usage = raw.usage_metadata or {}
        span.set_attribute("gen_ai.usage.input_tokens", usage.get("input_tokens", 0))
        span.set_attribute("gen_ai.usage.output_tokens", usage.get("output_tokens", 0))
        span.set_attribute("app.confidence", res.confidence if res else -1.0)
        return res

Then bucket the scores against labelled outcomes. A well-calibrated agent's accuracy tracks the bucket midpoint all the way down the scale, not just at the extremes.

Figure 2 — The loop that makes the number mean something. Every library in this article covers the first three boxes; nothing covers the last two, and that is where the work is.

The reason to draw it that way is that the two boxes nobody ships for you are the two that decide whether routing helps. Outcome capture needs a label, which usually means a human review queue, a downstream success signal, or a delayed ground truth like whether the ticket reopened. Threshold selection needs the bucket table and a decision about which error you would rather make. Everything upstream of those is configuration.

from collections import defaultdict

def calibration_report(records, n_bins=10):
    # records: list of (confidence, was_correct)
    bins = defaultdict(list)
    for conf, correct in records:
        bins[min(int(conf * n_bins), n_bins - 1)].append(correct)

    ece, total = 0.0, sum(len(v) for v in bins.values())
    for idx in sorted(bins):
        outcomes = bins[idx]
        midpoint = (idx + 0.5) / n_bins
        accuracy = sum(outcomes) / len(outcomes)
        ece += (len(outcomes) / total) * abs(accuracy - midpoint)
        print(f"conf {idx/n_bins:.1f}-{(idx+1)/n_bins:.1f}: "
              f"accuracy {accuracy:.2f} (n={len(outcomes)})")
    print(f"ECE ~ {ece:.3f}")

Temperature scaling, the single-parameter fix Guo et al. called "surprisingly effective" for classifier logits, does not transfer cleanly to a verbalized score from a hosted model, because you do not have the logits to scale. What you can do is fit a monotone mapping from reported score to observed accuracy on your own labelled set and route on the mapped value.

Where Fabriq Fits

Confidence routing decides whether an agent should act. It does not decide whether the agent is allowed to. Those are different questions and conflating them is how a 0.95 self-report ends up standing in for authorization. In the Agent Access Control Taxonomy a restriction the model applies to itself is aact:enforcement-point/in-prompt, which the file flags as bypassable by model error rather than a control.

Agentic Fabriq works on the second question. Credentials live in a per-user vault and are attached server-side at the moment of the call, so the agent's reasoning loop never holds the secret regardless of what it believes about itself. Every call carries two identities, the agent and the acting user, and lands an audit record with both. An agent's effective tool list is the intersection of the user's scopes and the agent's own, the taxonomy's aact:authorization-model/intersection-of-agent-and-user, so a low-confidence agent and a high-confidence one have exactly the same reach. Clearing a credential from the vault stops us using it; it is not the same as revoking the grant at the provider, and you should do that upstream too.

FAQ

Is AutoGen dead? Not dead, but in maintenance mode by Microsoft's own description, with no new features and community management. If you are starting now, the README points at Microsoft Agent Framework; if you want the old conversational API actively developed, that is AG2.

Should I use logprobs instead of asking the model? Probably not for a hosted instruction-tuned model. Tian et al. found verbalized confidence better calibrated than the models' own conditional probabilities, and several providers do not expose logprobs at all.

What threshold should I start with? Whatever you like, then throw it away. Log scores and outcomes for a week, build the bucket table, and set the threshold where accuracy actually falls off. A number copied from an article is a guess about someone else's model.

Does with_structured_output still work in LangChain 1.x? Yes. It is documented with no deprecation notice. What changed is that agents get structured output through response_format and a strategy object instead.

Does asking for a confidence score cost extra tokens? Barely on input, a little on output, and the output side is the one you can see. The score adds a field to the schema and a few tokens to the response, which is why include_raw=True and the usage_metadata it exposes are worth wiring up before you argue about it. Measure it on your own traffic rather than reasoning about it.

Can I get a confidence score without any of these libraries? Yes, and for a single call a schema and a JSON parse is genuinely enough. The libraries earn their place when you need the retry-on-invalid behaviour, the branch as a first-class object, or both.

Conclusion

The pattern has not changed since the first version of this article: make confidence a field in the output schema, branch on it with whatever mechanism the library already provides, and log it next to outcomes so you can check later whether it earned the trust you placed in it.

What changed is every API in the title. That is the actual lesson we would take from a year of drift. Write the wiring so the score is a schema field and a state key rather than a regex and a global threshold, because those two survive a framework migration and the rest of it does not.

Sources

All URLs read 29 September 2026.

• Structured output, LangChain docs — response_format, ToolStrategy and ProviderStrategy, handle_errors, the structured_response key, and the langchain>=1.1 / >=1.2 version floors.
• Models, LangChain docs — with_structured_output with no deprecation notice, include_raw returning raw/parsed/parsing_error, and the usage_metadata token fields.
• Middleware, LangChain docs — how middleware attaches to create_agent.
• Agent Access Control Taxonomy — the enforcement-point dimension, which asks where a decision is actually enforced, and the authorization-model term for an action allowed only if both the agent's policy and the user's permissions allow it.
• Built-in middleware, LangChain docs — RubricMiddleware for LLM-as-a-judge grading, plus the retry, fallback and call-limit middleware.
• Graph API, LangGraph docs — add_conditional_edges and its path map, START/END as the recommended entry and exit, set_entry_point as the legacy equivalent, Command for update-and-goto, and the 1,000-step recursion limit from version 1.0.6.
• AutoGen repository README — the maintenance-mode notice and the pointer to Microsoft Agent Framework.
• AutoGen to Microsoft Agent Framework migration guide — AssistantAgent single-turn versus Agent multi-turn by default, and the orchestration differences.
• AG2 quick start — the ag2 install name, the from autogen import ConversableAgent, LLMConfig path, and the run() then process() sequence.
• Instructor documentation — from_provider as the current constructor and response_model usage.
• Retrying, Instructor documentation — InstructorRetryException with n_attempts and failed_attempts, reask propagation, and Tenacity integration.
• Just Ask for Calibration (Tian et al., 2023) — verbalized confidence beating conditional probabilities, "often reducing the expected calibration error by a relative 50%".
• Can LLMs Express Their Uncertainty? (Xiong et al., 2023) — verbalized overconfidence, and no elicitation method consistently winning.
• Language Models (Mostly) Know What They Know (Kadavath et al., 2022) — calibration on multiple-choice formats, and the failure to generalize P(IK) to new tasks.
• On Calibration of Modern Neural Networks (Guo et al., 2017) — expected calibration error and temperature scaling.
• GenAI span semantic conventions, OpenTelemetry — attribute names and requirement levels for operation, provider, model and token usage, all at Development stability.
• PyPI release records — version and release dates for langchain, langgraph, ag2, autogen-agentchat, agent-framework and instructor as at 29 September 2026.