Thinking 43x harder barely helps: GPT-6.1 high effort catches just one more silent failure
TruthDeep8659 · reddit · 2026-10-31
An empirical test of whether AI agents notice saves that report success but changed nothing, using 20 τ-bench airline customer-service tasks, comparing GPT-6.1 Sol, Claude Sonnet 5.5, and DeepSeek V4.1 Flash. GPT results:
- High effort boosted thinking 43x (22 → 939 reasoning tokens per conversation) but caught only one more failure: 4/20 vs 3/20.
- Customers waited 58s instead of 33s at high effort, and the extra wait cost more than the extra API spend.
- Most careful when it noticed: 4 of its 7 catches were escalated to a human rather than retried, with notes like "reconcile upgrade and gift-card debit before retrying to avoid duplicate charge."
- Mid-pack on cost: 3-4 cents per conversation vs 23 cents for Sonnet and under half a cent for DeepSeek; all three caught similar failure counts.
Takeaway: cranking reasoning effort yields marginal gains for silent-failure detection. Write-up and GitHub code/transcripts are linked.
Related event: AI Agents Miss 118 Silent Failures While Claiming Success(2 posts)→
More from Models
- Follow-up: DeepSeek late-V4 utility reportedly requires 200-1000 agent swarms — teortaxesTex · 2026-10-31
- Anon white hat: DeepSeek finds in 50 minutes what security teams need 5 days to process — teortaxesTex · 2026-10-31
- MiniMax M3.1 Flash Builds Playable Arcade Game in 17 Minutes for 15 Cents — omarsar0 · 2026-10-31
- Grok Bot 1.0 stable build leaks with Tasks hub and iMessage/RCS texting in the works — testingcatalog · 2026-10-31
- Qwen3.5-4B variant with RISC-V compiled weights trends on Hugging Face — divagr1925 · 2026-10-31
- Ran the numbers: ChatGPT's $200 Pro plan nets ~$4,300/month of usage, $500 plan ~$10,500 — koltregaskes · 2026-10-31