What 4 calibration papers do (and don't) say about gating agent tool calls on model confidence
Xjjjjyn · reddit · 2026-11-17
- The author reviews four papers behind the "act when confidence > X, ask a human below X" pattern: Guo et al. 2017 (modern nets are poorly calibrated out of the box; temperature scaling helps), Kadavath et al. 2022 (large LLMs calibrate well on multiple-choice/true-false but struggle with knowing-what-they-know on new tasks), Tian et al. 2023 (verbalized confidence beats token probabilities, often cutting ECE by 50% relative), Xiong et al. 2023 (LLMs tend to be overconfident when verbalizing; calibration improves with capability).
- Key gap: all four measure answer correctness on QA-style benchmarks. None measures whether a multi-step tool call is the right action on your own data — a TriviaQA score says little about whether those 40 rows should be deleted.
- Design takeaways: (1) gate actions on computable-without-the-model properties — action type, reversibility, records touched, amount, audience; (2) use confidence only to route reads and reviews, never to authorize send/pay/delete/publish; (3) calibrate any score on your own labeled sample and recheck after every model change; (4) prefer multi-sample agreement over one self-reported number.
- Aligns with OWASP LLM06:2025 Excessive Agency (human approval for high-impact actions). Discloses affiliation with a commercial agent product and asks whether anyone has measured calibration on tool-call correctness post-model-upgrade.
Related event: Calibration literature review questions confidence gating for agent actions(2 posts)→
More from coding & agent
- Agent Substrate: open-source zero-trust runtime suspends idle agents, fits 10x more per hardware — Saboo_Shubham_ · 2026-11-17
- ChatGPT Wallet solves secure payment, not buying the right thing — krishnan · 2026-11-17
- Grok bot phones a store and secures a $1,511 refund for its user — elonmusk · 2026-11-17
- Practical tips to cut agent memory costs on OpenAI: cache TTL, Flex, fewer calls — gabrielchua · 2026-11-17
- 10 open-source repos to build an entire AI content factory solo — aftahi_ai · 2026-11-17
- Open-source 3D tools slash game-creature modeling from $12k to a prompt — ID_R_McGregor · 2026-11-17