GPT-5.6 Leads on DeepSWE While Cutting Costs
rohanpaul_ai · x · 2026-07-10
GPT-5.6 Sol scored between 72% and 73% on the DeepSWE coding agent benchmark, averaging about $8.40 per task. Claude/Fable-5 achieved a top score of around 70%, but with higher costs ranging from $13 to $22 per task.
The post also explains DeepSWE's methodology: it requires models to complete long software engineering tasks within real open-source repositories, going beyond simple code completion. The evaluation checks if the final patch is actually viable, testing repository navigation, bug diagnosis, code editing, tool usage, test execution, and multi-step debugging capabilities.
Related event: GPT-5.6 Tops DeepSWE Leaderboard with Superior Cost-Efficiency(11 posts)→
More from coding & agent
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
- Firecrawl offers 2.5x credits to a16z Speedrun founders, unveils $75M Series B — devdigest · 2026-10-27
- Chums: open-source Apple Watch app puts Claude Code and Codex on your wrist — brandon_galang · 2026-10-27
- Open-source Clairic Studio turns pasted text into typed data and working code in 5 languages — dfinke · 2026-10-27
- Matt Pocock shares expand/contract trick: split one-way-door PRs to de-risk big changes — mattpocockuk · 2026-10-27