Pinocchio: an external calibrator brings fast uncertainty estimates to black-box LLM APIs
micahgoldblum · x · 2026-10-02
Researchers from the Goldblum/Tom Goldstein group at UMD introduce Pinocchio, an external calibrator that predicts correctness of responses from black-box API LLMs that don't return log-probabilities or allow fine-tuning.
- Trained jointly on responses from 7 LLMs, it reaches 0.862 AUROC on held-out responses and zero-shot transfers to 13 unseen models across 8 organizations.
- Needs only a single forward pass, no access to logits, weights, or internal states; a 0.8B text-only checkpoint matches the largest model's AUROC.
- Paper, models, and code are released — integrating it into an existing repo takes two lines of code.
More from Research
- SYNTH paper finds epistemic calibration emerges in models from 300M parameters — cephaloform · 2026-10-02
- Nemotron 3 Ultra report reveals MOPD distillation teachers must share compatible training pipelines — cwolferesearch · 2026-10-02
- Sasha Rush publishes tutorial on sampling without randomness, centered on variance reduction — srush_nlp · 2026-10-02
- Self-attesting ledgers proposed as fix for missing shared baselines across AI labs — pratyusha_PS · 2026-10-02
- Researchers show AI models can "reproduce": mating by complementary strengths, no gradient descent — rvp · 2026-10-02
- MICCAI 2026 wraps up with BrainWorks and medical imaging workshops — PTenigma · 2026-10-02