Intuit paper: a 90%-accurate agent restart monitor can swing success by -5 to +35 points
intuit · hf · 2026-11-06
Intuit researchers show that classification accuracy alone doesn't determine whether restarting an AI agent improves success. Using an exact alarm-to-action identity for clean restarts, they prove a fixed 90%-accuracy monitor can swing success rates from a 5-point loss to a 35-point gain depending only on the replacement attempt's residual-budget success probability.
They introduce two decision-value evaluation techniques: a decision-sufficiency criterion (a binary-action specialization of Blackwell's comparison of experiments) and a finite-sample, model-free decision audit using simultaneous binomial intervals over existing traces without extra model calls.
On 194 externally evaluated executions across 19 SymPy issues from SWE-bench Verified, a 2.29-point apparent restart gain traced to a single surviving failed attempt, a second model lost 42.29 points, no positive replication on prospective cohorts, and zero of 103 live comparisons met the certification bar. Monitors should be judged by recovery opportunity, decision value, and statistical support.
More from coding & agent
- Expertise is knowing all the ways projects fail — that's what lets you babysit agents — RexDouglass · 2026-11-06
- Raindrop launches Simulations: auto-test AI agents on every PR with real production scenarios — soleio · 2026-11-06
- MoE Training Handbook Ch.2: A Runbook From Hardware Tests to Your First MoE — Nouamanetazi · 2026-11-06
- 9 copy-paste Claude prompts for code review, meeting notes, blind-spot checks and more — manishkhosiya · 2026-11-06
- RicursiveAI Releases From-Scratch Dense Ripple-65B, Tops 80% on Terminal Bench 2.1 — DZhang50 · 2026-11-06
- Team's 24/7 infra-monitoring agent spots anomalies 'before the fire,' user says — antoine_chaffin · 2026-11-06