How to Add Confidence Scores to Agent Outputs (So They Know When They Don't Know)

Overview

Take a support agent that pulls a customer's order history, drafts a refund justification, and hands it to a human for one click of approval. Most of the justifications are correct. Once in a while the agent cites a return-window exception that isn't in the policy anywhere, writes it in the same declarative sentence structure as everything else, and the approver, who has clicked "approve" a couple hundred times already today, clicks approve on that one too.

Nothing in the output marked that sentence as different from the others, and that's the actual gap. Fluency and knowledge are two different computations that happen to look identical on the page, so a model can produce a well-formed, confident sentence about something it flatly doesn't know. Humans signal the difference without thinking about it: I know this is right, I think so, but check me, I have no idea. Agents don't, not unless you build the signal in.

None of the four approaches below is a solved problem on its own: asking the model to grade itself, having a second model check the first, reading the shape of its own token probabilities, or combining them into something worth routing on. Each fails in a specific, learnable way, and knowing which failure you're looking at when a score comes back wrong is most of the engineering work.

Self-Rating Prompts (LLM Introspective Confidence)

The simplest method, and a surprisingly useful one, is to ask the model directly: rate your confidence from 0 to 1.

This works because models carry implicit uncertainty signals. When the probability mass behind a generation is spread across many plausible continuations, models tend to hedge in their language, and if you ask for a number explicitly, that hedging shows up as a lower self-score too. The catch, which we come back to under calibration, is that the relationship between a model's stated confidence and its actual accuracy is loose. It's still a useful first signal, and it costs nothing extra to collect.

Prompt Template

You are an expert assistant. Provide:
1. Your answer.
2. A confidence score between 0 and 1, representing how certain you are.

Format:
{
  "answer": "...",
  "confidence": 0.0-1.0
}

Python Implementation

A common mistake is to ask for JSON in the prompt and then call json.loads directly on the raw text. That works most of the time and fails the rest, usually when the model wraps the JSON in prose or a code fence. The more reliable approach is OpenAI's Structured Outputs, which constrain decoding to a schema you define so the response is guaranteed to parse:

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class AnswerWithConfidence(BaseModel):
    answer: str
    confidence: float  # 0.0 to 1.0

prompt = "What is the derivative of x^3 + 4x?"

completion = client.chat.completions.parse(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Provide your answer and a confidence score from 0 to 1."},
        {"role": "user", "content": prompt},
    ],
    response_format=AnswerWithConfidence,
)

result = completion.choices[0].message.parsed
print("Answer:", result.answer)
print("Confidence:", result.confidence)

Using parse with a Pydantic schema means you never defend against malformed JSON. If you're on a model or provider that doesn't support Structured Outputs, fall back to response_format={"type": "json_object"} and parse defensively.

Pros

• Trivial to add. No extra calls, no infrastructure.
• Works well on reasoning questions, where the model can introspect on its own logic.
• A reasonable input among several, rather than a standalone verdict.

Cons

• Poorly calibrated out of the box. Models are frequently overconfident, and scores cluster near the top of the range.
• Reflects how the model talks about certainty, not its measured error rate. Treat it as a soft signal until you calibrate it.

Cross-Model Comparison (A Checker Model Scores the Primary)

A sturdier method is to look for disagreement between two models. When two models with different training and different failure modes disagree, the odds that at least one of them is wrong go up fast.

That mirrors ensemble methods in classical machine learning, where variance across independent estimators is itself a signal worth reading. There are two common framings here: have a second model independently answer and check for agreement, or have it act as a critic and score the first model's answer. The critic pattern is cheaper to wire up, so it's the one we show.

Architecture

• A capable primary model (for example GPT-4.1, Claude Sonnet 4, or Llama 3.1 405B) produces the answer.
• A faster, cheaper secondary model (for example GPT-4o-mini, Llama 3.1 8B, or Phi-3-mini) critiques and scores it.

Picking a secondary from a different model family is deliberate. Two models trained on overlapping data and aligned with similar techniques tend to share blind spots. A checker from a different lineage is more likely to catch the kind of error the primary is prone to.

Implementation

import json

primary = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Explain quantum entanglement."}],
)

answer = primary.choices[0].message.content

checker_prompt = f"""
Evaluate the factual accuracy of the following answer.
Score it from 0 (clearly wrong) to 1 (fully correct and well supported).

Answer:
{answer}

Return a JSON object: {{"score": number}}
"""

secondary = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": checker_prompt}],
    response_format={"type": "json_object"},
)

score = float(json.loads(secondary.choices[0].message.content)["score"])
print("Cross-model confidence:", score)

Pros

• Tracks factual correctness better than a single model's self-report.
• Cuts single-model blind spots, especially with a checker from a different family.

Cons

• Adds latency and cost. At minimum one extra inference call per answer.
• Can still fail when both models are confidently wrong about the same thing, which is most likely on a widely repeated misconception.

Entropy-Based Uncertainty (Token-Level Probability)

If your model or API exposes logprobs, you can compute token entropy: a signal pulled straight from the model's own probability distribution rather than from anything it says about itself.

High entropy means probability mass is scattered across many candidate tokens, which points to low confidence. Low entropy means the distribution is sharp and concentrated on one token, which points to high confidence.

Entropy Formula

For a single token position, given candidate probabilities p_i drawn from the model's distribution, the Shannon entropy is:

Two practical details matter. First, the OpenAI Chat Completions API only returns the top few candidates per position (top_logprobs accepts an integer from 0 to 20), so you're computing entropy over a truncated distribution, not the full vocabulary. Renormalizing those probabilities to sum to 1 keeps the number stable. Second, entropy is most informative on the tokens that actually carry the answer, a number, a name, a yes or no, rather than on filler words, so it's worth aggregating with some care rather than averaging blindly.

Python Implementation

import math
from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1",
    logprobs=True,
    top_logprobs=5,
    messages=[{"role": "user", "content": "What is the capital of Peru?"}],
)

content = response.choices[0].logprobs.content

def token_entropy(top_logprobs):
    # Renormalize the truncated top-k distribution so the
    # returned probabilities sum to 1 before measuring entropy.
    probs = [math.exp(t.logprob) for t in top_logprobs]
    total = sum(probs) or 1.0
    probs = [p / total for p in probs]
    return -sum(p * math.log(p) for p in probs if p > 0)

# Average per-token entropy across the generated answer.
entropies = [token_entropy(tok.top_logprobs) for tok in content]
avg_entropy = sum(entropies) / len(entropies)

# Map entropy onto a 0-1 confidence score. log(k) is the maximum
# possible entropy for k candidates, so this stays in range.
max_entropy = math.log(len(content[0].top_logprobs))
confidence = max(0.0, 1.0 - avg_entropy / max_entropy)
print("Entropy-based confidence:", confidence)

This is a deliberately conservative heuristic, not a calibrated probability. It's grounded in something real, though: the model's actual output distribution, which is why it tends to track correctness better than a self-reported number does.

Pros

• The most mathematically grounded of the three signals.
• Reflects the model's genuine internal uncertainty.

Cons

• Requires logprob access, which many hosted APIs, including OpenAI's newer Responses API, don't currently expose.
• Needs careful normalization across token types. Raw per-token entropy is noisy on filler tokens.

Using Confidence Thresholds to Trigger Fallback Logic

Once your agent has a score, you can turn it into policy. Thresholds let the agent act differently depending on how sure it is: high enough, act on the answer; middling, ask a clarifying question first; low, escalate or re-route; very low, trigger a full retry or a cross-model check.

Code Example

def route_based_on_confidence(answer, confidence):
    if confidence > 0.8:
        return answer
    if 0.5 < confidence <= 0.8:
        return "Before I continue, I need to ask: can you clarify what you mean?"
    if 0.3 < confidence <= 0.5:
        return "I'm not fully confident. Let me double-check using another model."
    return "I'm not confident in this answer. Escalating to a human or retrying."

The exact cutoffs aren't sacred. They should track the cost of being wrong. Start with sensible defaults, log every routing decision, and tighten the bands once you've seen how the scores distribute against real outcomes.

Handling "I'm Not Sure" Paths Cleanly

Refusing outright is the lazy version of handling low confidence. The better version hands the user a concrete next step instead of an apology.

Here's a clean fallback script that combines safety with decent UX:

I'm not confident that I can answer this correctly.
To help you better, I can:
1. Ask a clarifying question,
2. Gather external data using approved tools, or
3. Escalate to a human reviewer.
Which would you prefer?

Python wrapper

def uncertainty_response() -> str:
    return (
        "I'm not sure I can answer that with high confidence.\n\n"
        "I can:\n"
        "- Ask a clarifying question\n"
        "- Check additional sources\n"
        "- Escalate to a human operator\n\n"
        "What would you like me to do?"
    )

Why this matters

Admitting uncertainty tends to build trust rather than erode it. We think of it as calibrated trust: a user relies on a system in proportion to how reliable it actually is, and that only works if the system signals when it's unsure. An agent that confidently asserts a wrong answer trains users to distrust everything, or worse, to trust everything. An agent that flags its own low-confidence moments lets a user spend their attention where it's actually needed.

Calibrating the Scores So the Numbers Mean Something

A confidence score is only useful if a high score actually corresponds to being right most of the time. Raw model self-ratings rarely clear that bar. They tend to be overconfident and bunched near the top of the scale. Calibration is the step that turns a relative signal into one you can attach a threshold to and trust.

The cheapest way to check calibration is a reliability diagram: bucket your predictions by stated confidence, then plot the average stated confidence in each bucket against the actual accuracy in that bucket. A perfectly calibrated agent sits on the diagonal. Most agents start well above it: high claimed confidence on answers that are, in practice, right noticeably less often. The summary number for this is Expected Calibration Error, the average gap between confidence and accuracy across buckets.

def expected_calibration_error(records, n_bins=10):
    # records: list of (confidence, was_correct) from a labeled eval set
    bins = [[] for _ in range(n_bins)]
    for conf, correct in records:
        idx = min(int(conf * n_bins), n_bins - 1)
        bins[idx].append((conf, correct))

    total = len(records)
    ece = 0.0
    for b in bins:
        if not b:
            continue
        avg_conf = sum(c for c, _ in b) / len(b)
        accuracy = sum(int(ok) for _, ok in b) / len(b)
        ece += (len(b) / total) * abs(avg_conf - accuracy)
    return ece

Once you can measure the gap, you can correct it. Two lightweight options cover most needs. The first is temperature scaling on entropy-derived scores: fit a single scalar that sharpens or softens the distribution until the buckets line up with the diagonal. The second is isotonic regression over a labeled evaluation set, which learns a monotonic mapping from raw scores to calibrated probabilities without assuming any particular shape. Both run on a modest set of labeled examples, and both let you keep the simple thresholds from the previous section while making the numbers behind them honest.

Putting It All Together: A Full Confidence Pipeline

Below is a minimal end-to-end pipeline that combines the signals built up above: self-rating from the primary model, cross-model critique from a cheaper checker, token-level entropy, threshold-based routing, and a clean fallback when nothing clears the bar.

Figure 1 — Three independent signals feed one calibrated threshold. None of the three is trustworthy alone; the combiner and the calibration step are what make the number worth routing on.

The three signals are averaged below for clarity, but a simple average is the weakest possible combiner. In practice you'd weight each signal by how well it predicts correctness on your own data, often learned with a small logistic regression over the three inputs, and feed the result through the calibration step above before applying thresholds.

import json
import math
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class AnswerWithConfidence(BaseModel):
    answer: str
    confidence: float

def token_entropy(top_logprobs):
    probs = [math.exp(t.logprob) for t in top_logprobs]
    total = sum(probs) or 1.0
    probs = [p / total for p in probs]
    return -sum(p * math.log(p) for p in probs if p > 0)

def get_answer_with_confidence(question):
    # 1. Primary answer with self-rating (schema-constrained) + logprobs.
    primary = client.chat.completions.parse(
        model="gpt-4.1",
        logprobs=True,
        top_logprobs=5,
        messages=[
            {"role": "system", "content": "Answer, then give confidence 0 to 1."},
            {"role": "user", "content": question},
        ],
        response_format=AnswerWithConfidence,
    )
    parsed = primary.choices[0].message.parsed
    answer = parsed.answer
    self_conf = parsed.confidence

    # 2. Entropy-based confidence from the generated tokens.
    content = primary.choices[0].logprobs.content
    entropies = [token_entropy(tok.top_logprobs) for tok in content]
    avg_entropy = sum(entropies) / len(entropies)
    max_entropy = math.log(len(content[0].top_logprobs))
    entropy_conf = max(0.0, 1.0 - avg_entropy / max_entropy)

    # 3. Cross-model critique from a cheaper, different model.
    critique_prompt = f'''Rate factual accuracy 0-1. Answer: {answer}
Return JSON: {{"score": number}}'''
    critic = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": critique_prompt}],
        response_format={"type": "json_object"},
    )
    critic_conf = float(json.loads(critic.choices[0].message.content)["score"])

    # 4. Aggregate (replace with a calibrated, weighted combiner in prod).
    final_conf = (self_conf + entropy_conf + critic_conf) / 3

    # 5. Threshold routing.
    if final_conf > 0.8:
        return {"answer": answer, "confidence": final_conf}
    if final_conf > 0.5:
        return {"answer": f"I might be wrong: {answer}", "confidence": final_conf}
    return {
        "answer": "I'm not confident enough. Let me re-check or clarify.",
        "confidence": final_conf,
    }

Wire this behind every consequential agent action and the failure mode changes entirely. Instead of failing silently with full confidence, the agent degrades gracefully: it asks, it defers, it escalates.

Conclusion

None of the four techniques above is a confidence score in the sense a statistician would recognize until you calibrate it. Self-rating tells you how the model talks about certainty. Entropy tells you how sharp its token distribution was. A checker model tells you whether a second, differently trained system agrees. Each is a partial view, and averaging three partial views is still a partial view, just a better one.

What actually changes an agent's behavior isn't picking the single best signal. It's wiring whatever combination you land on into a threshold that branches the control flow, logging the outcome of every routing decision, and checking every so often whether a high score still means what you think it means. That last habit is the one it's easiest to skip, and the one that makes the other three worth doing at all.