Reading logits beats asking: token-probability confidence hits AUROC 0.77 vs 0.66 for verbalised

LLMs Are Already Confidence-Aware Zero-Shot Classifiers

Owen Cook, Xingyi Song

cs.CL, cs.AI, cs.LG

2026-11-19

Reading token logits beats asking the model for confidence: AUROC 0.77 vs 0.66 across 11 open-weight LLMs, and decision-only scoring cuts decode steps by 88-96%.

What problem this solves

Using a zero-shot LLM as a text classifier is routine now: labelling data, intent detection, content moderation. The label alone is half the product; the other half is how much to trust it. If the model says 60% confidence when right and 95% when wrong, the classifier cannot sit in front of a human-review pipeline.

There are several ways to pull confidence out of an LLM: ask it to state its confidence in words (verbalised confidence), sample multiple answers and count agreement, ask "is this correct?" after the fact (P(True)), or score candidate labels with token probabilities. These methods had rarely been compared under matched conditions, and most prior work focused on question answering, where uncertainty comes from recalling knowledge. In classification, uncertainty mostly reflects task ambiguity. Two researchers at the University of Sheffield ran the comparison and added a cheaper scoring rule along the way.

Method

Eleven open-weight models from 1B to 32B parameters (Gemma 4, Llama 3.2, Qwen 2.5/3/3.5), five datasets (Banking77 with 77 classes, CLINC150 with 151, 20 Newsgroups, SST-5, FEVER with 3), all zero-shot, no chain-of-thought, greedy decoding. The datasets were chosen because their labels share decoding paths: in SST-5, "very positive" and "very negative" both start with "very", which is exactly where multi-token label scoring tends to break.

Black-box side: JSON output with confidence stated before or after the label, as a percentage or a five-level Likert scale; sampling with 10 runs at temperature 0.5, majority proportion as confidence; and P(True), which asks the model whether its own answer holds and reads the first-token probability.

White-box side: all labels go into a trie (prefix tree); at each decoding step the full vocabulary is softmaxed and invalid tokens masked, so output stays inside the label set. Four aggregation rules are compared: total log-likelihood, length normalisation by token count, Google's machine-translation length penalty with α = 0.6, and a probability tree that softmaxes sibling nodes at each step and multiplies along the path.

The new piece is decision-only scoring. Most tokens on a path through the trie are forced, with only one valid continuation, so their likelihoods add noise proportional to label length. Decision-only scoring touches only decision tokens, the positions where labels actually diverge, and skips everything forced. This cuts compute and removes some of the bias that length normalisation introduces.

Results

Accuracy barely moves across confidence methods (means all around 0.72; the aggregate table drops the two smallest Llama 3.2 models). The differences show up in confidence quality:

MethodAUROC (discrimination)ECE (calibration)Note
Logit scoring (constrained variants)0.74-0.770.185-0.223Best discrimination overall
Verbalised (Likert)0.606-0.6360.150-0.167Best calibration, weak discrimination
Sampling (10 runs, t=0.5)0.5690.254Loses on both
P(True)0.7410.227Keeps free generation, still discriminates

(AUROC measures how well confidence separates correct answers from wrong ones, 0.5 being chance; ECE measures how far stated confidence drifts from actual accuracy.)

Decision-only scoring cuts decode steps by 88-96% on the three largest label sets (Banking77 96%, CLINC150 92%, 20NG 88%) with scores matching full-label scoring. Replacing labels with single-token identifiers (A/B/C) preserves discrimination but costs 3-6 accuracy points on average.

Against the commercial Jev 1.13.0 classifier (a TypeSafe AI model that returns a full distribution over labels): Gemma 4 31B with constrained JSON and decision-only scoring reaches 0.802 accuracy and 0.775 AUROC, beating Jev's 0.788 accuracy while trailing its 0.801 AUROC. Qwen 3.5 9B with length normalisation and decision-only scoring reaches 0.793 AUROC and 0.053 ECE, a lower ECE than Jev's 0.086. On raw calibration Jev still wins overall (best top-label Brier at 0.134).

One practical detail: smaller models produce more format errors, with Gemma 4 E2B failing to parse 0.22% of outputs against 0.01% for the 31B model. Constrained decoding eliminates format errors but can distort the output distribution, so whether to use it depends on the model.

Why it matters

The takeaway is directly usable. If an LLM handles zero-shot classification or semi-automated labelling for you, do not ask it how confident it is; read the logits. Verbalised confidence looks calibrated but cannot separate correct answers from wrong ones, so a "route low-confidence cases to humans" pipeline built on it will wave through plenty of errors. Decision-only scoring makes scoring a 151-class label set affordable. Open-weight models with no extra training reach commercial-classifier territory, which matters wherever privacy rules out an API. This is a method-selection paper, not a new model, and its value is measuring scattered recipes under one ruler.

Limitations

The authors list their own: Jev's architecture and training data are undocumented, so contamination with these public datasets cannot be ruled out, and the same holds for the open-weight models; temperature scaling was not tried, though fitting it on a small labelled set would likely improve ECE for logit methods; everything is zero-shot with one prompt per method-dataset pair, so prompt sensitivity is untested; all datasets are English; sampling ran at a single setting (10 samples, temperature 0.5).

Three things stand out on a close read. The 88-96% reduction is in decode steps, not wall-clock time, and the authors note overheads eat into it. Good discrimination with weak calibration means logit confidence values should not be used as probabilities without a calibration step first. And the aggregate table drops the two smallest Llama 3.2 models without explanation in the text.

Terms

Source

What people are saying

Related papers

All paper explainers