Five frontier models all solved the same bugs, but cost varied 14x and Claude refused 40%
PromptPhanter · reddit · 2026-07-23
Frontier coding-agent benchmark across five models
A Reddit post summarizes a benchmark of five frontier models from OpenAI and Anthropic on a small JavaScript bug-fix task suite.
- Setup: 18 tasks, 410 total runs, four tools (list/read/write files, run tests), and a 24-step budget on default settings.
- With tests available: every model solved every task, including multi-file bugs, but the cost per solved task varied by 14x — from about $0.01/fix for the cheapest model to $0.14/fix for the most expensive.
- With tests hidden: the models split sharply on one bug. The two GPT-5.6 variants solved it 13/13, while every Claude model managed 5/13 or fewer. The pooled Fisher’s exact test was p ≈ 3×10⁻⁹.
- The author’s interpretation: GPT models tended to make the smallest fix that preserved behavior, while Claude models often rewrote surrounding code in ways that broke hidden contracts.
- One Claude model, Fable 5, refused 29 of 82 runs with a cyber-related refusal label on ordinary bug-fix requests — roughly a 40% refusal rate.
The writeup, methodology, raw data, and refusal probes are published in the linked blog post and GitHub repo for reproduction or extension to other models.
More from coding & agent
- Creator shares a Codex + ChatCut workflow for turning long videos into platform clips — yihui_indie · 2026-07-23
- Claude Code’s CLAUDE.md is only context; hard rules need PreToolUse hooks — thisdudelikesAI · 2026-07-23
- Claude Code’s /context and /memory commands reveal which memory files are actually loaded — thisdudelikesAI · 2026-07-23
- Add Claude Code rules only after the same mistake shows up twice — thisdudelikesAI · 2026-07-23
- Claude Code’s auto memory writes project notes to disk and loads only the first 200 lines — thisdudelikesAI · 2026-07-23
- Claude Code supports @path imports in CLAUDE.md to reuse shared rules across repos — thisdudelikesAI · 2026-07-23