Five frontier models all solved the same bugs, but cost varied 14x and Claude refused 40%

PromptPhanter · reddit · 2026-07-23

Frontier coding-agent benchmark across five models

A Reddit post summarizes a benchmark of five frontier models from OpenAI and Anthropic on a small JavaScript bug-fix task suite.

Setup: 18 tasks, 410 total runs, four tools (list/read/write files, run tests), and a 24-step budget on default settings.
With tests available: every model solved every task, including multi-file bugs, but the cost per solved task varied by 14x — from about $0.01/fix for the cheapest model to $0.14/fix for the most expensive.
With tests hidden: the models split sharply on one bug. The two GPT-5.6 variants solved it 13/13, while every Claude model managed 5/13 or fewer. The pooled Fisher’s exact test was p ≈ 3×10⁻⁹.
The author’s interpretation: GPT models tended to make the smallest fix that preserved behavior, while Claude models often rewrote surrounding code in ways that broke hidden contracts.
One Claude model, Fable 5, refused 29 of 82 runs with a cyber-related refusal label on ordinary bug-fix requests — roughly a 40% refusal rate.

The writeup, methodology, raw data, and refusal probes are published in the linked blog post and GitHub repo for reproduction or extension to other models.

Original post →

Five frontier models all solved the same bugs, but cost varied 14x and Claude refused 40%

More from coding & agent