Five frontier models all solved the same bugs, but cost varied 14x and Claude refused 40%
PromptPhanter · reddit · 2026-07-23
Frontier coding-agent benchmark across five models
A Reddit post summarizes a benchmark of five frontier models from OpenAI and Anthropic on a small JavaScript bug-fix task suite.
- Setup: 18 tasks, 410 total runs, four tools (list/read/write files, run tests), and a 24-step budget on default settings.
- With tests available: every model solved every task, including multi-file bugs, but the cost per solved task varied by 14x — from about $0.01/fix for the cheapest model to $0.14/fix for the most expensive.
- With tests hidden: the models split sharply on one bug. The two GPT-5.6 variants solved it 13/13, while every Claude model managed 5/13 or fewer. The pooled Fisher’s exact test was p ≈ 3×10⁻⁹.
- The author’s interpretation: GPT models tended to make the smallest fix that preserved behavior, while Claude models often rewrote surrounding code in ways that broke hidden contracts.
- One Claude model, Fable 5, refused 29 of 82 runs with a cyber-related refusal label on ordinary bug-fix requests — roughly a 40% refusal rate.
The writeup, methodology, raw data, and refusal probes are published in the linked blog post and GitHub repo for reproduction or extension to other models.
More from coding & agent
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11