Evaluating LLM Confidence: The Calibration Methodology
Logprob aggregation, semantic entropy, Brier score, and Platt scaling: the methodology for a calibrated confidence score you can act on.
When a model says it is 0.92 confident, I do not treat that number as a probability. It is a generated sentence, chosen the same way every other token is chosen, and it has no direct link to whether the answer is correct. I stopped using it as a gate after seeing how often a stated 0.9 sat on top of an accuracy closer to 0.65 on a labeled set.
The reason is structural, not a quirk of one model. The model was trained to predict the next token, not to predict whether its own answer is right, and training on human feedback rewards answers that sound decisive over answers that hedge. So the stated number lands well above actual accuracy in the range where it matters most, the high-confidence range where you are most likely to act on it without checking.
A confidence score I actually trust comes from three things rather than one: a signal that has a real relationship to correctness, a measurement of how far that signal sits from the truth on a labeled set, and a calibration step that maps the raw signal onto a usable probability. This is how I run each, and how the whole loop sits in production.
Why the model’s stated confidence is not a probability
There are three plain reasons the verbal number is biased, and none of them are fixed by a better prompt.
- The model predicts tokens, not its own correctness. Its internal token probabilities are calibrated for the next-token task, but the sentence “I am 90 percent sure” is just a phrase it learned to produce, not a readout of how likely the claim is.
- Training on human feedback rewards decisiveness, so the model hedges less than it should. Across the major model families, the stated confidence tracks accuracy loosely but sits systematically too high in the 0.7 to 1.0 band.
- The self-report is produced after the answer, conditioned on the prompt’s wording. Change the wording and the number changes, which is a sign it is not measuring anything internal.
So I keep the verbal number as one weak signal in an ensemble and never as the score itself.
Two signals that actually relate to correctness
There are two signals yoau can get without training anything, and which one you use depends on whether the provider exposes token probabilities.
Get Rishav Hada’s stories in your inbox
Join Medium for free to get updates from this writer.
When the API exposes logprobs, which most open-weight servers do and some commercial endpoints do on request, I use two derived signals:
- The average probability of the chosen token across the tokens that carry meaning, the entities, numbers, and names, rather than across filler words like “the” and “of,” which sit near certain regardless of correctness.
- The spread of the next-token distribution at each of those positions. When the model was choosing between several plausible tokens on a claim, that hesitation correlates with error.
When the API hides logprobs, I estimate uncertainty from how much the model disagrees with itself:
- Sample several answers at a moderate temperature, group the ones that mean the same thing, and measure the spread across meaning-groups rather than across wordings. Grouping by meaning is the important part, since “Paris is the capital” and “The capital is Paris” are the same answer in different words.
- Grouping can be done with a small entailment model or by embedding the answers and clustering them. The published result is that this meaning-based spread detects hallucinations 10 to 15 points more accurately than counting surface-string disagreement or trusting the stated confidence.
Two caveats keep these honest:
- For reasoning models that hide their thinking, the visible logprobs only cover the final answer, which is often a confident summary of an uncertain process. Treat logprob aggregation as a secondary signal there.
- Meaning-based spread is misleading on open-ended tasks where many answers are correct, because the spread is high for a good reason. Anchor it on tasks that have a defensible right answer.
Measure how far the signal sits from the truth
A raw signal is only useful once you know how miscalibrated it is, and that requires a labeled set.
- Brier score is the headline number: the mean squared error between the predicted probability and the actual correct-or-not outcome. It rewards both calibration, predictions matching observed accuracy, and sharpness, the willingness to commit rather than hover at 0.5.
- Expected Calibration Error is the more readable companion: bin the predictions, compare each bin’s predicted confidence to its actual accuracy, and average the gaps. Under 0.05 on your labeled set is a reasonable production target.
- The reliability diagram is the picture both summarize, predicted confidence against observed accuracy, where a trustworthy signal sits on the diagonal and most off-the-shelf models sit above it in the high-confidence range.
The labeled set is the part I do not skip. I pull 500 to 2,000 real production traces rather than invented test cases, skew the sample toward the hardest tenth of inputs, and hand-label them. Everything downstream depends on this set being representative.
Calibrate the signal into a usable probability
The raw signal, whether verbalized, logprob-based, or spread-based, is rarely calibrated out of the box. Two standard post-hoc fixes map it onto a probability you can act on.
- Platt scaling fits a simple logistic curve on pairs of (raw signal, was it correct). It is the default starting point, stable from a couple hundred examples, and it works when the miscalibration is smooth and roughly linear. It commonly cuts calibration error by 30 to 60 percent on verbalized confidence alone.
- Isotonic regression fits a flexible monotonic curve instead, which handles odd-shaped miscalibration but needs more data, on the order of a thousand examples, to stay stable. I reach for it when Platt scaling under-corrects in the tails.
Fit the calibrator on one split, confirm Brier and ECE actually improve on a held-out split, and then deploy it as a thin wrapper on the raw signal. Recalibrate when the model version changes, when the prompt template changes meaningfully, or when the input distribution shifts, because the calibrator is a contract between the model and the traffic and it breaks when either side moves.
Put the loop into production
Three pieces carry this from a notebook into a running system.
- Capture the signals on every span. Record the logprob-based signal, the meaning-based spread, and the stated confidence as attributes on the trace alongside the normal request fields, so calibration runs on real traffic rather than a sample.
- Keep the labeled set current and re-measure Brier and ECE against it on a schedule, so you notice drift as a number rather than as a customer complaint.
- Gate on the calibrated probability, not the raw signal. Combine the signals, map them through the calibrator, and escalate to a human when the calibrated probability falls below a threshold. The threshold is a business decision, since medical triage and internal chat sit at different points on the curve, while the calibration underneath it is engineering.
The cost worth planning for is the sampling. Estimating meaning-based spread means several generations per call, so route only the high-stakes calls through it and keep the cheap single-call path for the rest.
What to remember
- The model’s stated confidence is one signal and never the primary one. Use logprob aggregation when the API exposes token probabilities and meaning-based spread when it does not, and combine them.
- Brier score and ECE on a held-out labeled set are the contract. Without the labeled set you do not know how far off the raw signal is, and any threshold you pick is a guess.
- Post-hoc calibration is cheap, and you have to repeat it. A simple calibrator on a few hundred examples often cuts calibration error by half, and you refit it whenever the model, the prompt, or the distribution moves.
