How to Add Confidence Scores to Agent Outputs (So They Know When They Don't Know)

Overview

Take a support agent that pulls a customer's order history, drafts a refund justification, and hands it to a human for one click of approval. Most of the justifications are correct. Once in a while the agent cites a return-window exception that isn't in the policy anywhere, in the same declarative sentence structure as everything else, and the approver, who has clicked "approve" two hundred times already today, clicks approve on that one too.

Nothing marked that sentence as different, and that's the gap. Fluency and knowledge are two different computations that happen to look identical on the page. Humans signal the difference without thinking about it: I know this is right, I think so, but check me, I have no idea. Agents don't, not unless you build the signal in.

A confidence signal can come from only three places, and the engineering, the cost and the failure modes all follow from which one you pick. You can read what the model says about itself, read the distribution it sampled from, or sample more than once and measure how much the answers agree.

Figure 1 — The three vantage points. Self-report and token entropy are free with the answer you already generated; agreement and a checker model each cost extra inference.

None of them is solved on its own. Each fails in a specific, learnable way, and knowing which failure you're looking at is most of the work.

Self-Rating Prompts (LLM Introspective Confidence)

The simplest method is to ask the model directly: rate your confidence from 0 to 1.

It works better than it looks like it should. Kadavath et al. found that "larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format", and that models produce a usable P(True) for their own answers, improving when they can see several candidate samples first. The same paper found the weak spot: a model's estimate that it knows an answer at all transfers poorly to a task it wasn't calibrated on.

Tian et al. pushed further on models fine-tuned with human feedback. Across ChatGPT, GPT-4 and Claude on TriviaQA, SciQ and TruthfulQA they report that "verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities", cutting expected calibration error by roughly half in relative terms. If you assumed the number pulled from logprobs must beat the number the model types, sit with that result a moment.

The catch, which we come back to under calibration, is that better is not good. Xiong et al. put the same class of method through five datasets and five models and found that "LLMs, when verbalizing their confidence, tend to be overconfident", with black-box methods landing between 0.522 and 0.605 AUROC on failure prediction.

Prompt Template

You are an expert assistant. Provide:
1. Your answer.
2. A confidence score between 0 and 1, representing how certain you are.

Format:
{
  "answer": "...",
  "confidence": 0.0-1.0
}

Python Implementation

A common mistake is to ask for JSON in the prompt and then call json.loads on the raw text. That works most of the time and fails the rest, usually when the model wraps the JSON in prose or a fence. Structured Outputs is the fix: OpenAI documents it as "ensures the model will always generate responses that adhere to your supplied JSON Schema", with a refusal field instead of schema-shaped output when the model declines.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class AnswerWithConfidence(BaseModel):
    answer: str
    confidence: float  # 0.0 to 1.0

prompt = "What is the derivative of x^3 + 4x?"

completion = client.chat.completions.parse(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Provide your answer and a confidence score from 0 to 1."},
        {"role": "user", "content": prompt},
    ],
    response_format=AnswerWithConfidence,
)

result = completion.choices[0].message.parsed
print("Answer:", result.answer)
print("Confidence:", result.confidence)

Using parse with a Pydantic schema means you never defend against malformed JSON. If you're on a model or provider without schema-constrained decoding, fall back to a JSON-object response format and parse defensively.

Cross-Model Comparison (A Checker Model Scores the Primary)

A sturdier method is to look for disagreement between two models. When two models with different training and different failure modes disagree, the odds that at least one of them is wrong go up fast.

Two framings are common: a second model answers independently and you check for agreement, or it acts as a critic and scores the first answer. The critic pattern is cheaper to wire up, so it's the one we show.

Architecture

• A capable primary model (for example GPT-4.1, Claude Sonnet 4, or Llama 3.1 405B) produces the answer.
• A faster, cheaper secondary model (for example GPT-4o-mini, Llama 3.1 8B, or Phi-3-mini) critiques and scores it.

Picking the secondary from a different family is not a stylistic preference. Panickssery et al. showed that LLM evaluators recognise their own output and score it higher than human annotators do, and that a model's ability to identify its own text correlates linearly with the size of that bias. Use the primary as its own judge and you measure familiarity as much as correctness.

Implementation

import json

primary = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Explain quantum entanglement."}],
)

answer = primary.choices[0].message.content

checker_prompt = f"""
Evaluate the factual accuracy of the following answer.
Score it from 0 (clearly wrong) to 1 (fully correct and well supported).

Answer:
{answer}

Return a JSON object: {{"score": number}}
"""

secondary = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": checker_prompt}],
    response_format={"type": "json_object"},
)

score = float(json.loads(secondary.choices[0].message.content)["score"])
print("Cross-model confidence:", score)

A close relative needs no second model at all. SelfCheckGPT samples the same model several times and checks the samples against each other: a model asked about something it knows says roughly the same thing each time, while a fabrication drifts. Manakul et al. built it for the case where you have neither an external knowledge base nor access to model internals, which is most hosted APIs.

Entropy-Based Uncertainty (Token-Level Probability)

If your provider returns logprobs, you can compute token entropy: a signal pulled straight from the model's own probability distribution rather than from anything it says about itself.

High entropy means probability mass scattered across many candidates, which points to low confidence. Low entropy means a sharp distribution concentrated on one token.

Availability is the first thing to check, and it varies more than people expect:

Table 1 — Logprob availability, checked against each provider's own API reference. An entropy-based design is not portable across these three; a verbalized or agreement-based one is.

Entropy Formula

For a single token position, given candidate probabilities p_i drawn from the model's distribution, the Shannon entropy is:

Two details matter. You only get the top few candidates per position, capped at 20 on Chat Completions, so the entropy is over a truncated distribution; renormalizing to sum to 1 keeps it stable. And entropy is informative on the tokens carrying the answer, a number, a name, a yes or no, not on filler, so aggregate with care rather than averaging blindly.

Python Implementation

import math
from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1",
    logprobs=True,
    top_logprobs=5,
    messages=[{"role": "user", "content": "What is the capital of Peru?"}],
)

content = response.choices[0].logprobs.content

def token_entropy(top_logprobs):
    # Renormalize the truncated top-k distribution so the
    # returned probabilities sum to 1 before measuring entropy.
    probs = [math.exp(t.logprob) for t in top_logprobs]
    total = sum(probs) or 1.0
    probs = [p / total for p in probs]
    return -sum(p * math.log(p) for p in probs if p > 0)

# Average per-token entropy across the generated answer.
entropies = [token_entropy(tok.top_logprobs) for tok in content]
avg_entropy = sum(entropies) / len(entropies)

# Map entropy onto a 0-1 confidence score. log(k) is the maximum
# possible entropy for k candidates, so this stays in range.
max_entropy = math.log(len(content[0].top_logprobs))
confidence = max(0.0, 1.0 - avg_entropy / max_entropy)
print("Entropy-based confidence:", confidence)

This is a conservative heuristic, not a calibrated probability, and it has a structural blind spot Kuhn, Gal and Farquhar named precisely: token entropy punishes a model for having several ways to say the same true thing, because "different sentences can mean the same thing" while their token sequences do not. Their fix, semantic entropy, samples several answers, clusters them by meaning using entailment rather than string match, and measures entropy over the clusters. It predicts accuracy better than token-level baselines.

You pay for it in calls. Kossen et al. put the overhead at a "5-to-10-fold increase in computation cost", which is why they went looking for an approximation and found one in the model's hidden states, cutting overhead "to almost zero". That route is open only if you run the weights yourself.

Which Signal to Use, and When

Most write-ups leave this decision implicit. Cost below is extra model calls per answer, beyond the one you were making anyway.

Table 2 — Pick by constraint, not by elegance. If your provider returns no logprobs, two of these five rows are closed to you before you start.

Self-consistency was not designed as a confidence signal and works as one anyway. Wang et al. introduced it to improve accuracy by sampling several reasoning paths and taking the most common answer, reporting gains of 17.9 points on GSM8K, 12.2 on AQuA, 11.0 on SVAMP, 6.4 on StrategyQA and 3.9 on ARC-challenge. The distribution you marginalise over is itself the uncertainty estimate, free with a computation you may already run.

Using Confidence Thresholds to Trigger Fallback Logic

Once your agent has a score, you can turn it into behavior. Thresholds let the agent act differently depending on how sure it is: high enough, act on the answer; middling, ask a clarifying question first; low, escalate or re-route; very low, trigger a full retry or a cross-model check.

Code Example

def route_based_on_confidence(answer, confidence):
    if confidence > 0.8:
        return answer
    if 0.5 < confidence <= 0.8:
        return "Before I continue, I need to ask: can you clarify what you mean?"
    if 0.3 < confidence <= 0.5:
        return "I'm not fully confident. Let me double-check using another model."
    return "I'm not confident in this answer. Escalating to a human or retrying."

The cutoffs aren't sacred; they should track the cost of being wrong. The formal version is selective prediction, where the metric is the accuracy you keep at a given coverage rather than accuracy alone. Chen et al. tuned a model for that and moved AUACC on CoQA from 91.23% to 92.63%, AUROC from 74.61% to 80.25%. That's the shape of gain to expect: real, and smaller than a demo suggests. Start with defaults, log every routing decision, and tighten the bands once you see how scores distribute against outcomes.

Handling "I'm Not Sure" Paths Cleanly

Refusing outright is the lazy version of handling low confidence. The better version hands the user a concrete next step instead of an apology, which is aact:failure-behavior/refuse-and-explain in the Agent Access Control Taxonomy rather than a guessed result. A fallback script that does both:

I'm not confident that I can answer this correctly.
To help you better, I can:
1. Ask a clarifying question,
2. Gather external data using approved tools, or
3. Escalate to a human reviewer.
Which would you prefer?

Python wrapper

def uncertainty_response() -> str:
    return (
        "I'm not sure I can answer that with high confidence.\n\n"
        "I can:\n"
        "- Ask a clarifying question\n"
        "- Check additional sources\n"
        "- Escalate to a human operator\n\n"
        "What would you like me to do?"
    )

Admitting uncertainty tends to build trust rather than erode it, but be honest about the mechanism. Buçinca et al. tested interventions that make a person engage with an AI suggestion instead of accepting it, across 199 participants, and found cognitive forcing "significantly reduced overreliance compared to the simple explainable AI approaches". Participants also rated those interfaces less favourably. An uncertainty prompt is a small cognitive forcing function, and it buys error-catching with some satisfaction score. We think that's a trade worth making anywhere the agent writes to a system of record.

Calibrating the Scores So the Numbers Mean Something

A confidence score is only useful if a high score actually corresponds to being right most of the time. Raw model self-ratings rarely clear that bar out of the box.

The cheapest way to check is a reliability diagram: bucket your predictions by stated confidence, then plot the average stated confidence in each bucket against the actual accuracy in that bucket. A perfectly calibrated agent sits on the diagonal. Most start well above it. The summary number is Expected Calibration Error, the average gap between confidence and accuracy across buckets.

def expected_calibration_error(records, n_bins=10):
    # records: list of (confidence, was_correct) from a labeled eval set
    bins = [[] for _ in range(n_bins)]
    for conf, correct in records:
        idx = min(int(conf * n_bins), n_bins - 1)
        bins[idx].append((conf, correct))

    total = len(records)
    ece = 0.0
    for b in bins:
        if not b:
            continue
        avg_conf = sum(c for c, _ in b) / len(b)
        accuracy = sum(int(ok) for _, ok in b) / len(b)
        ece += (len(b) / total) * abs(avg_conf - accuracy)
    return ece

Treat that number with suspicion even as you track it. Nixon et al. concluded that expected calibration error "has numerous flaws which we outline", including instability in binning: change the bin count and the ranking between two methods can flip. Adaptive binning, where each bucket holds an equal number of predictions rather than an equal slice of the range, is the stabler variant. Report the bin count next to the error.

Once you can measure the gap you can correct it. Temperature scaling is the standard move and long predates language models: Guo et al. found modern networks badly miscalibrated and this single-parameter variant of Platt scaling "surprisingly effective". Isotonic regression is the alternative, learning a monotonic mapping from raw scores to probabilities without assuming a shape. Both run on a modest labeled set, and both let you keep simple thresholds while making the numbers behind them honest.

How Each Signal Fails, and What the Symptom Looks Like

Scores go wrong in patterned ways. Five are worth recognising on sight.

Everything is 0.9. Plot a histogram of self-rated scores and you'll find a spike at 0.9, another at 0.95, and nothing below 0.7. The model is picking round numbers that read as confident, which is the overconfidence Xiong et al. measured. The symptom is a threshold that never fires. The fix is calibration, not a sterner prompt.

Entropy says unsure, the answer is right. A question with five correct phrasings produces high token entropy at the first content token. If your low-confidence bucket is full of correct answers in varied wording, you're measuring lexical choice, not doubt.

The checker always agrees. If your critic scores 0.9 on nearly everything, check whether it shares a family with the primary. Swap in a different lineage and see whether the distribution widens. If it does, you were reading self-preference.

The score tracks length. Averaging entropy across every token means a long answer full of connective tissue scores as confident and a terse one doesn't. Aggregate over the answer-bearing tokens, or take the maximum rather than the mean.

It was calibrated, then a model upgrade landed. Calibration is fitted to a model, a prompt and a task distribution. Change any of the three and the mapping is stale. Kadavath et al. found a model's own sense of whether it knows something transfers poorly to new tasks, and so does the scalar you fitted on top. Treat thresholds as versioned config.

Putting It All Together: A Full Confidence Pipeline

Below is a minimal end-to-end pipeline combining the signals built up above: self-rating from the primary model, cross-model critique from a cheaper checker, token-level entropy, threshold-based routing, and a clean fallback when nothing clears the bar.

Figure 2 — Three independent signals feed one calibrated threshold. None of the three is trustworthy alone; the combiner and the calibration step are what make the number worth routing on.

The three signals are averaged below for clarity, and a simple average is the weakest possible combiner. Weight each signal by how well it predicts correctness on your own data, usually a small logistic regression over the three inputs, then send the result through the calibration step before thresholding.

import json
import math
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class AnswerWithConfidence(BaseModel):
    answer: str
    confidence: float

def token_entropy(top_logprobs):
    probs = [math.exp(t.logprob) for t in top_logprobs]
    total = sum(probs) or 1.0
    probs = [p / total for p in probs]
    return -sum(p * math.log(p) for p in probs if p > 0)

def get_answer_with_confidence(question):
    # 1. Primary answer with self-rating (schema-constrained) + logprobs.
    primary = client.chat.completions.parse(
        model="gpt-4.1",
        logprobs=True,
        top_logprobs=5,
        messages=[
            {"role": "system", "content": "Answer, then give confidence 0 to 1."},
            {"role": "user", "content": question},
        ],
        response_format=AnswerWithConfidence,
    )
    parsed = primary.choices[0].message.parsed
    answer = parsed.answer
    self_conf = parsed.confidence

    # 2. Entropy-based confidence from the generated tokens.
    content = primary.choices[0].logprobs.content
    entropies = [token_entropy(tok.top_logprobs) for tok in content]
    avg_entropy = sum(entropies) / len(entropies)
    max_entropy = math.log(len(content[0].top_logprobs))
    entropy_conf = max(0.0, 1.0 - avg_entropy / max_entropy)

    # 3. Cross-model critique from a cheaper, different model.
    critique_prompt = f'''Rate factual accuracy 0-1. Answer: {answer}
Return JSON: {{"score": number}}'''
    critic = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": critique_prompt}],
        response_format={"type": "json_object"},
    )
    critic_conf = float(json.loads(critic.choices[0].message.content)["score"])

    # 4. Aggregate (replace with a calibrated, weighted combiner in prod).
    final_conf = (self_conf + entropy_conf + critic_conf) / 3

    # 5. Threshold routing.
    if final_conf > 0.8:
        return {"answer": answer, "confidence": final_conf}
    if final_conf > 0.5:
        return {"answer": f"I might be wrong: {answer}", "confidence": final_conf}
    return {
        "answer": "I'm not confident enough. Let me re-check or clarify.",
        "confidence": final_conf,
    }

Log final_conf, each component score, the branch taken and the eventual outcome. Without that table you have no reliability diagram, no way to weight the combiner, and no evidence when someone asks whether the score does anything.

Frequently Asked Questions

How do I get a confidence score from the OpenAI API? Two ways, and they compose. Set logprobs: true with top_logprobs between 0 and 20 on Chat Completions and compute entropy over the returned candidates, or ask the model for a number and constrain the shape with Structured Outputs, which OpenAI documents as always adhering to your JSON Schema. On the Responses API the token probabilities arrive when you request message.output_text.logprobs in the include array.

Does Claude return logprobs? No. The Messages API reference documents no logprobs or top_logprobs parameter, so an entropy-based design does not port to it. Verbalized confidence, self-consistency across samples, and a checker model all work there.

Is asking a model to rate its own confidence reliable? More reliable than its token probabilities on RLHF-tuned models, per Tian et al., and still overconfident in absolute terms, per Xiong et al. Both are true at once. Treat it as a signal that needs a calibration layer, not as a probability.

What's a good confidence threshold for an agent? There isn't a portable number: the threshold encodes the cost of being wrong, not a property of the model. Pick the coverage you need, find the score that delivers it on a labeled set, re-derive after any model change.

What's the difference between token entropy and semantic entropy? Token entropy measures spread over strings, so two correct paraphrases look like disagreement. Semantic entropy clusters sampled answers by meaning first, then measures spread over clusters. It's the better signal on free-form output and costs five to ten times as much to compute.

How do I know my calibration is still good? Keep a labeled slice of production traffic and redraw the reliability diagram weekly, reporting the bin count alongside the error. A sudden drop in error is usually a distribution shift, not an improvement.

Conclusion

None of these techniques is a confidence score in the sense a statistician would recognize until you calibrate it. Self-rating tells you how the model talks about certainty. Entropy tells you how sharp its token distribution was, over strings rather than meanings. A checker model tells you whether a second system agrees, unless it's the same system wearing a different name.

What changes an agent's behavior isn't picking the best signal. It's wiring whatever combination your provider permits into a threshold that branches the control flow, logging the outcome of every routing decision, and checking whether a high score still means what you think it means. That last habit is the easiest to skip, and the one that makes the other three worth doing.

Sources

All URLs read 2026-09-29.

• Agent Access Control Taxonomy — the failure-behavior dimension, where declining and explaining is its own term, and the human-oversight dimension, which separates post-hoc review from approval on a risk rule.
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — Tian et al.; verbalized confidence better calibrated than conditional probabilities on ChatGPT, GPT-4 and Claude across TriviaQA, SciQ and TruthfulQA, with a relative 50% ECE reduction.
• Language Models (Mostly) Know What They Know — Kadavath et al., Anthropic; calibration on multiple-choice and true/false formats, P(True) over multiple samples, and P(IK) failing to transfer to new tasks.
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs — Xiong et al., ICLR 2024; verbalized overconfidence, the prompting/sampling/aggregation framework, and the 0.522–0.605 AUROC band for black-box methods.
• Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation — Kuhn, Gal and Farquhar, ICLR 2023; semantic equivalence as the flaw in token-level entropy, and entropy over meaning clusters as the fix.
• Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs — Kossen et al.; the 5-to-10-fold compute cost of semantic entropy, and approximating it from the hidden states of a single generation.
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models — Manakul, Liusie and Gales; sampling-based consistency checking with no external database and no access to model internals.
• Self-Consistency Improves Chain of Thought Reasoning in Language Models — Wang et al., ICLR 2023; the sampled-path vote and the reported gains of 17.9 on GSM8K, 12.2 on AQuA, 11.0 on SVAMP, 6.4 on StrategyQA and 3.9 on ARC-challenge.
• LLM Evaluators Recognize and Favor Their Own Generations — Panickssery, Bowman and Feng; self-preference bias and its linear relationship with a model's ability to recognise its own text.
• Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs — Chen et al., EMNLP 2023 Findings; selective prediction framing and the CoQA move from 91.23% to 92.63% AUACC and 74.61% to 80.25% AUROC.
• On Calibration of Modern Neural Networks — Guo et al., ICML 2017; modern networks are poorly calibrated, and temperature scaling as a single-parameter variant of Platt scaling.
• Measuring Calibration in Deep Learning — Nixon et al.; the flaws in expected calibration error and the stability advantage of adaptive binning.
• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making — Buçinca, Malaya and Gajos, CSCW 2021; N=199, reduced overreliance, and participants preferring the interfaces that reduced it least.
• Chat Completions API reference, OpenAI — logprobs returning per-token log probabilities and top_logprobs as an integer between 0 and 20.
• Responses API reference, OpenAI — requesting message.output_text.logprobs through the include parameter.
• Structured Outputs, OpenAI — schema adherence guarantees, the refusal field, and first-request schema latency.
• Messages API reference, Anthropic — the full request body parameter list, which contains no logprobs option.