SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
Yuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He, Lei Zhang, Yue Liu, Zelin Zhao, Terry Yue Zhuo, Jialun Cao, Siyu Ye, Tianyu Liu, Kai Cai, Shing-Chi Cheung, Xiaodong Gu
COLM 2026
cs.CL, cs.SE
2026-08-11
170 expert-curated multilingual refactoring tasks averaging 11 files and 262 lines each. GPT-5.2 tops out at 41.2% under OpenHands; open-weight models reach within five points at a twentieth of the cost.
AI coding agents have improved fast enough that resolve rates on SWE-bench Verified now exceed 75%, and the gap between top systems keeps shrinking. Rising scores do not prove the agents got better. OpenAI audited SWE-bench Verified and found flawed tests in nearly 60% of unsolved instances: some tests are too narrow and reject correct solutions, others are too broad and check requirements the task never stated. Frontier models can even reproduce gold patches verbatim from training data. OpenAI deprecated the benchmark for these reasons.
Saturation is one problem; task type is another. Existing benchmarks mostly test single-file bug fixes and feature work. The activity that most stresses cross-file coordination, refactoring, is underserved. Refactoring means changing code in bulk while preserving its external behavior. Both OpenAI and Cursor call project-scale refactors a core long-horizon use case, yet no benchmark actually measured it.
SWE-Bench ProMax keeps 170 instances from 29,782 candidates, spanning seven languages (Python, Java, TypeScript, Go, C, C++, Rust) and 70 repositories. Curation runs in three stages. Stage 1 uses the GitHub API to find repositories with at least 500 stars, an approved license, and a primary language among the seven, then pulls commits after January 2025 whose messages contain "refactor" but not "bug fix". Stage 2 builds a Docker environment for each commit, applies the gold patch, and runs the full test suite; anything that fails is dropped. Stage 3 is human-plus-LLM rewriting: analyze the commit, drop tasks that are too simple or single-file, remove overly narrow and overly broad tests, rewrite terse commit messages into precise self-contained specifications, and verify by hand.
The verification bar is strict: the specification must be both necessary and sufficient for the gold patch. The surviving tasks average 11.4 source files and 261.6 lines of code; the largest touches 244 files in NASA's F'Prime flight software framework.
Six frontier models were tested under two scaffolds (mini-swe-agent and OpenHands), capped at 300 steps and $10 per instance. The best, GPT-5.2, reaches only 41.2% under OpenHands, far below the 75%+ on SWE-bench Verified.
| Model | Scaffold | Resolve | Cost/instance |
| GPT-5.2 | OpenHands | 41.2% | $3.60 |
| Claude Sonnet 4.6 | OpenHands | 38.8% | $4.77 |
| GLM-5 | OpenHands | 36.5% | $0.24 |
| Qwen3.5 | OpenHands | 36.5% | $0.78 |
| GPT-5.2 | mini-swe-agent | 21.8% | $0.19 |
Three things stand out. Open-weight models reach within five points of closed models at a twentieth of the cost: GLM-5 hits 36.5% for $0.24 while Claude Sonnet 4.6 costs $4.77 for 38.8%. The scaffold matters enormously, with GPT-5.2 jumping from 21.8% to 41.2% moving from mini-swe-agent to OpenHands. No model dominates every language; Claude leads TypeScript and Rust, GPT-5.2 leads Python and C.
The value here is diagnostic, not the ranking. First, large-scale refactoring is where agents genuinely stall, and it is among the most common jobs in real engineering. Second, the failure mode is consistent: agents do not fail to find the files, they fail to finish. They correctly edit the core files but stop short of pushing the same transformation to peripheral call sites, docs, configs, and test fixtures, so downstream inconsistencies break the tests. Failed runs consume more rounds than successful ones, stuck in read-edit-test-revert loops. The bottleneck is cross-file coordination, not single-step reasoning.
The sample is small at 170. TypeScript's 28 instances come largely from Angular (25 of 28), so language-level conclusions for TypeScript are narrow. The tasks are nominally behavior-preserving refactors, but the authors' own classification shows 41.2% also involve bug fixes and 43.5% new features, so "pure refactoring" is not a clean label. Only two scaffolds were tested, and the cost figures are point-in-time snapshots.