SWE Refactor Bench: Only 5.4% of 520 Runs Pass, Best Model Opus 5 Scores 47/100
YouJiacheng · x · 2026-08-26
A new benchmark, SWE Refactor Bench (arXiv:2608.23564), tests whether coding agents can autonomously complete long-horizon, whole-repository migrations to pay down technical debt.
Existing benchmarks only check behavioral correctness, enabling an easy hack—copying the original implementation to pass tests—termed Blindness by the authors. The benchmark includes 20 whole-repo migrations across 4 kinds of technical debt, with a three-stage protocol: (1) Migration Audit verifies the migration actually occurred; (2) Behavioral Tests measure correctness with a fixed suite; (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioral differences.
Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 (5.4%) pass all three stages; 13 of 20 tasks receive no accepted solution; the best model, claude-opus-5, scores just 47.0/100. The paper argues migration completeness and behavioral correctness are distinct abilities.
Related event: SWE Refactor Bench: Coding Agents Pass Only 5.4% of Repo-Wide Migrations(2 posts)→
More from coding & agent
- How to debug MCP tool call failures and cascading errors in production? — Prod_whiz · 2026-08-26
- Snowflake reveals Agent hidden costs, reducing trial cost by 33%-45% — StasBekman · 2026-08-26
- Hermes supports gateway multiplexing to save resources at cost of isolation — alexcovo_eth · 2026-08-26
- Voice AI in 2026: Why real-time voice agents are hard to ship in production — Latent Space · 2026-08-26
- Parker Conrad: AI Agents So Good We Don't Need a Buyout from Silverlake — andreisavu · 2026-08-26
- Hermes Agent adds 44 new tool connectors for one-click access — NousResearch · 2026-08-26