Agents Say 'Done' When Nothing Happened: 97 of 118 Silent Failures Went Uncaught
TruthDeep8659 · reddit · 2026-10-31
A small study using 20 airline customer-service tasks from τ-bench tested Claude Sonnet 5.5, GPT-6.1 Sol, and DeepSeek V4.1 Flash at low and high reasoning effort on scenarios where a save reports 'success' but nothing changed:
- Agents told the customer it was done in 97 of 118 failed saves
- 29 customers received confirmation numbers for bookings that didn't exist
- High reasoning effort caught 11 failures vs 10 for low effort — essentially no difference
- No agent ever re-checked whether a save worked; every catch was luck from a later step
Takeaway: thinking harder doesn't fix silent failures — workflows need explicit verification steps. Write-up, code, and transcripts are public.
Related event: AI Agents Miss 118 Silent Failures While Claiming Success(2 posts)→
More from coding & agent
- PentestCode Review: A Terminal AI Agent for Pentesting and Bug Bounty — glenbeer · 2026-10-31
- Unanswerable questions inflate Recall@K: dev open-sources RAG evaluator to fix it — infrixo · 2026-10-31
- Yacine's advice to new programmers: push hardware to the limit, bitbang GPIO — yacineMTB · 2026-10-31
- Resin: experimental Rust LLM inference library vibes-code onto Intel HPU in hours — charles_irl · 2026-10-31
- Uncensored-model agent swarms for red-teaming, and a DePIN data comeback thesis — 0xJeff · 2026-10-31
- AI social network Spawn runs on an autonomous Claude agent, burned $100K in month one — TAbrodi · 2026-10-31