Paper Benchmarks Black-Box vs White-Box LLM Confidence Estimation Across 11 Open-Weight Models
kalyan_kpl · x · 2026-11-21
A new paper examines how LLMs can perform zero-shot text classification while also estimating prediction confidence—without any fine-tuning—to address the problem of knowing when to trust model outputs.
- Approach: systematically compares black-box methods (verbalised confidence, repeated sampling) against white-box methods (token probabilities, P(True), logit-based scoring); such methods had not been systematically benchmarked for classification before.
- Setup: evaluates 11 open-weight LLMs (1B–32B parameters) across five datasets—Banking77, CLINC150, 20 Newsgroups, SST-5, and FEVER—using accuracy, Macro-F1, AUROC, and Expected Calibration Error (ECE).
The takeaway: LLMs already work as confidence-aware zero-shot classifiers out of the box, but calibration quality varies substantially across confidence estimation methods.
More from Research
- AnyS1: a 4B System-1 model cuts agent workflow costs by 75–91% without losing accuracy — kalyan_kpl · 2026-11-22
- MLatom 3.27 ships the whole AIQM quantum chemistry family with one pip install, AIQM3 now Apache 2.0 open source — PavloDral · 2026-11-22
- Uncensored Inkling 2 Mini v2: self-distillation lifts useful answers from 52% to 79% with open pipeline — Turbulent-Sky5396 · 2026-11-22
- Looped transformers explained: Inkling 2 Mini loops 12 middle layers 3x for deeper reasoning without more weights — Prompt Engineering · 2026-11-22
- Researcher claims Gemini Neon helped him land a potential Nature-worthy physics idea — xennygrimmato_ · 2026-11-22
- TU Delft open-sources City4CFD for watertight 3D city geometry in urban flow simulations — Promptmethus · 2026-11-22