Four calibration papers say model confidence shouldn't gate risky agent tool calls
Xjjjjyn · reddit · 2026-11-17
Reviewing four calibration papers before trusting "act when confidence > X" agent safety heuristics:
What they report
- Guo et al. 2017: modern nets are poorly calibrated out of the box; temperature scaling fixes much of it (pre-LLM classifiers)
- Kadavath et al. 2022: large models are well calibrated on formatted multiple-choice/true-false, but struggle with "do I know this" on new tasks
- Tian et al. 2023: verbalized confidence beats token probabilities for RLHF chat models, often cutting ECE by a relative 50%
- Xiong et al. 2023: LLMs tend to be overconfident when verbalizing; calibration improves with capability; sample-consistency is a viable signal
The gap: all four measure answer correctness on QA benchmarks — none measures whether a multi-step tool call is the right action on your data. A TriviaQA score says nothing about "are these the right 40 rows to delete."
Design choices that follow (authors' own, not paper findings):
- Gate on computable attributes: action type, undoability, audience, records touched, amount
- Confidence routes reads and review only — never grants send/pay/delete/publish permission
- Calibrate on your own labeled data and recheck after every model change
- Prefer cross-sample agreement over a single self-reported score
Consistent with OWASP LLM06:2025 (Excessive Agency). Author discloses affiliation with a commercial agent product.
Related event: Calibration literature review questions confidence gating for agent actions(2 posts)→
More from coding & agent
- Agent Substrate: open-source zero-trust runtime suspends idle agents, fits 10x more per hardware — Saboo_Shubham_ · 2026-11-17
- ChatGPT Wallet solves secure payment, not buying the right thing — krishnan · 2026-11-17
- Grok bot phones a store and secures a $1,511 refund for its user — elonmusk · 2026-11-17
- Practical tips to cut agent memory costs on OpenAI: cache TTL, Flex, fewer calls — gabrielchua · 2026-11-17
- 10 open-source repos to build an entire AI content factory solo — aftahi_ai · 2026-11-17
- What 4 calibration papers do (and don't) say about gating agent tool calls on model confidence — Xjjjjyn · 2026-11-17