SAP paper teaches LLM agents to predict their own failure (AUROC 0.85)
SAP · hf · 2026-10-14
SAP researchers propose outcome-grounded confidence calibration: an LLM agent's verbalized confidence is trained (under GRPO) against the eventual success of a multi-step episode rather than single-step correctness. Offline RL on a Qwen2.5-7B agent across ALFWorld, TextWorld and WebShop lifts per-step AUROC from near chance to 0.68-0.85, with the top-10% most-confident steps succeeding 4.3x more often. Online RL on live ALFWorld cuts expected calibration error from 0.42 to below 0.12 and improves selective execution without hurting task success. A new evaluation protocol separates episode depth, difficulty, and attempt-level foresight: the trained agent spots doomed tasks from step one, though distinguishing succeeding vs failing attempts on the same task remains harder. Useful for runtime control decisions: continue, verify, or hand off to humans.
More from coding & agent
- Dev runs Codex as a shell tool inside Claude for code review, skipping tmux hassles — abhi9u · 2026-10-14
- Using AI agents turns out to be accidental management training — dr_alphalyrae · 2026-10-14
- Rerunning the single-task experiment across all ipython harness versions produced the benchmark plot — yb2698 · 2026-10-14
- Small-model agent harnesses are fragile to environment noise — open-source ipython harness released — yb2698 · 2026-10-14
- For small models, ipython is all you need: single-tool harness ties production-grade scaffolds on TB-Lite — yb2698 · 2026-10-14
- 20-trial repeat experiments: OpenHands stays most stable across sampling settings — yb2698 · 2026-10-14