Comparing LLM Confidence Estimation Methods
crazed_vagus · reddit · 2026-07-16
This post compares different methods for LLM confidence estimation to determine which is best.
Background
- LLM judges are widely used for automating complex judgments.
- Combining "judgment results" with "confidence levels" enables active learning, prompt optimization, low-confidence sample filtering, and safety classification for agents/chatbots.
- The author notes this is still an early-stage research question, with both white-box and black-box methods competing.
Method Categories
- Text-based Methods:
- Directly asking the model "how sure are you."
- Inferring from uncertainty signals in the model's wording.
- Estimating difficulty and confidence based on reasoning length.
- Token-based Methods:
- Looking at the probability distribution of answer tokens.
- Using the yes/no probability of "Is this statement correct?" to estimate confidence.
- The author argues these methods have degraded in utility for reasoning models around 2026, as chain-of-thought contaminates the probability distribution.
- Sampling-based Methods:
- Gauging confidence by sampling multiple times and checking for self-consistency.
Article Claims
- The article will compare 8 black-box methods against 1 white-box method.
- The focus isn't on a single technique, but on how different methods vary in usability, applicable model types, and adaptability in the era of reasoning models.
Related event: Comprehensive Evaluation of LLM Confidence Estimation Methods(3 posts)→
More from Research
- Animation shows how an MLP’s first-layer weights change while learning MNIST — CatAstro_Piyush · 2026-07-22
- Project APE finds verifier reliability drops when papers contain multiple errors — soumitrashukla9 · 2026-07-22
- Project APE says verifier costs fell about 90x in a year as Chinese open models lead — soumitrashukla9 · 2026-07-22
- OpenAI-linked paper says capability RL can make models more reward-seeking — MariusHobbhahn · 2026-07-22
- Project APE builds its verifier benchmark from 100 AI-written papers with injected errors — soumitrashukla9 · 2026-07-22
- Paper proposes a CRED taxonomy and benchmark to measure research-error detectors — soumitrashukla9 · 2026-07-22