Four calibration papers reviewed: model confidence can't gate risky agent tool calls

Xjjjjyn · reddit · 2026-11-17

Before trusting "act when confidence > X, ask a human below X" for agent safety, the author reviews four calibration papers: Guo 2017 (NNs poorly calibrated, temperature scaling helps), Kadavath 2022 (large LLMs calibrate well on multiple-choice but struggle with knowing-what-they-know on new tasks), Tian 2023 (verbalized confidence beats token probabilities, often cutting ECE by 50% relative), Xiong 2023 (LLMs overconfident; consistency sampling helps). Key gap: all measure answer correctness on QA benchmarks, none measures whether a multi-step tool call is right on your data. Design takeaways: gate on model-free rules (action type, reversibility, blast radius, amount), use confidence only to route reads, recalibrate on your own labels after every model change, prefer multi-sample agreement. Aligns with OWASP LLM06:2025 Excessive Agency.

Related event: Calibration Literature Review: Model Confidence Unfit to Gate Risky Agent Actions(3 posts)→

Original post →

More from coding & agent

coding & agent channel →