Four calibration papers reviewed: model confidence can't gate risky agent tool calls
Xjjjjyn · reddit · 2026-11-17
Before trusting "act when confidence > X, ask a human below X" for agent safety, the author reviews four calibration papers: Guo 2017 (NNs poorly calibrated, temperature scaling helps), Kadavath 2022 (large LLMs calibrate well on multiple-choice but struggle with knowing-what-they-know on new tasks), Tian 2023 (verbalized confidence beats token probabilities, often cutting ECE by 50% relative), Xiong 2023 (LLMs overconfident; consistency sampling helps). Key gap: all measure answer correctness on QA benchmarks, none measures whether a multi-step tool call is right on your data. Design takeaways: gate on model-free rules (action type, reversibility, blast radius, amount), use confidence only to route reads, recalibrate on your own labels after every model change, prefer multi-sample agreement. Aligns with OWASP LLM06:2025 Excessive Agency.
More from coding & agent
- Pageloop Launches a Docs Agent That Auto-Captures Screenshots and Updates Stale Docs — prasanna_says · 2026-11-17
- OutfitterBench tests LLM agents on 200-day rental business, exposing commitment failures — Jie Huang · 2026-11-17
- Craftax framework evolves code policies via feedback cascade, reaching final stage 50% of time — Víctor Gallego · 2026-11-17
- Entity disambiguation: sentence context hits 97.9% vs 63.5% for name matching — YvesMulkers · 2026-11-17
- XState v6 Beta Launches: State Machines Rewritten for the AI Agent Era — mattpocockuk · 2026-11-17
- Matt Pocock shares a prompt for having AI agents evaluate new tech via draft PRs — mattpocockuk · 2026-11-17