SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, Qinhuai Na
cs.CL, cs.AI, cs.SE
2026-08-25
SWE Refactor Bench scores 20 whole-repo migrations in three stages. Of 520 runs, only 28 (5.4%) pass; claude-opus-5 at xhigh leads at 47.0/100.
SWE-bench-style evaluation lives on a red-to-green jump: a test fails before the patch and passes after it. Whole-repository stack migration has no such jump. The starting repo already passes its tests, so handing it back unchanged also scores a perfect suite. The paper names this Blindness: a behaviour-only scorer can award full marks to a repository that never moved.
Navers Lab (Einsia.AI) and Tsinghua University introduce SWE Refactor Bench: 20 real open-source repositories, spanning language, framework, platform, and build-toolchain debt. The job is to land the whole repo on a named target stack, with the old stack gone from source and from the build closure, and with observable behaviour identical. Tasks include cmark C to Rust, SQLite POSIX to WASI, and GraphHopper Dropwizard to Spring Boot. Combined implementation source is about 867k lines. Each task gives the agent 6 to 30 hours, offline, no network.
The agent gets a buildable State A, a State B declaration with pinned versions, an offline image that also carries the original toolchain, and a time budget. The evaluation suite lives in a separate image that is never mounted into the agent's container.
Scoring multiplies. Fail Stage I and the score is 0. Stage II is all-or-nothing. Clearing both floors the score at 0.4; each of the six verifiers that finds nothing adds 0.1. Survive all six and the score is 1.0.
Eight frontier models, 26 model-effort configurations, each configuration once on all 20 tasks: 520 scored runs. GPT-series models run under Codex; the other six under Claude Code.
| Best config | Score | Accepted | Cost / task |
| claude-opus-5 xhigh | 47.0 | 5/20 | $74.9 |
| gpt-5.6-sol max | 28.5 | 4/20 | $143.5 |
| kimi-k3 max | 19.5 | 2/20 | $28.9 |
| claude-sonnet-5 medium | 15.0 | 1/20 | $11.9 |
Of 520 runs, 340 (65.4%) pass Stage I and 118 (22.7%) pass every fixed check, but only 88 do both and reach Stage III. 28 (5.4%) survive all six verifiers. 13 of 20 tasks never receive an accepted solution. Mean score across all 520 is 13.44/100; among the 88 that reached Stage III it is 79.43. The hard part is getting to Stage III at all.
Completing the migration and preserving behaviour are distinct. 30 runs keep every fixed check by skipping the migration and die at Stage I. 252 complete the migration and break behaviour, dying at Stage II. cmark C to Rust is the sharp case: five runs go green on 4,184 checks, all of them Blindness. The Rust restates the C control flow statement for statement; ownership was never redesigned, and the original's manual memory management moved into unsafe Rust. acorn JavaScript to Rust is the mirror: 20 of 26 runs pass Stage I, and not one fills the fixed suite.
Among the 340 runs that did migrate, 58% reach 99% of the fixed checks and only 26% reach 100%. That last 1% is production-shaped. On Conduit Vue to React, four models all stop at 21768/21769: the original uses hash routing, so visiting / settles at /#/, while the React build stays at / and every bookmark breaks. On PyCryptodome setuptools to Meson, five models stop at 380/381: the wheel's METADATA long description is 0 characters, so the PyPI page would be blank.
Of the 88 submissions that missed no fixed check, 60 (68.2%) still take a counterexample within the hour; median time to a break is 17.0 minutes. ChartMuseum Gin to chi is the kind of miss a suite author cannot write in advance: the original truncates Content-Type at the first space or semicolon, the migrated version only at the semicolon, and one extra space before the boundary routes the request to a different upload handler.
By debt class, build-toolchain rewrites score 31.4 and language rewrites score 5.6. Build-toolchain has the highest Stage I pass rate (80.8%) and the lowest Stage III survival (17.6%). Framework rewrites invert that: 18.9% at Stage II, 56.0% at Stage III, and 14 of the 28 acceptances.
gpt-5.6-luna, DeepSeek V4 Flash, and GLM 5.2 accept nothing. Judged by the fixed suite alone, each would post a few perfect scores. Under the three-stage protocol they solve zero tasks.
For people who build coding agents, the paper splits "the tests went green" from "the migration happened." SWE-bench's red-to-green signal does not exist here: the start state is already green. Adding more behavioural checks cannot close Blindness, because the untouched implementation passes those checks by construction.
For teams that actually need to move a repository, the numbers are blunt. Current agents can grind most of a suite to 99% and still fail to ship a drop-in replacement. Swapping a build toolchain is comparatively tractable; rewriting a language almost is not. Giving the agent 30 hours and spending over a hundred dollars of API budget per task does not close that gap.
This is an evaluation paper, not a new migration algorithm. The contribution is making Blindness a hard gate, and sending other agents after the behaviours the suite author never thought to write.
There is no dedicated Limitations section. The boundaries that are in the text:
The set is 20 tasks, chosen by picking the debt first and the repository second. The authors say the results measure a gap on these 20 tasks; they do not rank the intrinsic difficulty of all migration projects. Each configuration runs each task once, so scores have no error bars. Agents work offline with no network; real migrations usually involve docs and issue trackers. Git history is squashed to a single commit so agents cannot read a prior migration out of git log.
Stage I is judged by gpt-5.6-sol. Against humans on 156 runs the agreement is 89.7% (κ=0.795), with 14 too-strict and 2 too-lenient calls. Of the 14 too-strict cases, 12 would have failed Stage II anyway; the judge let no disguised submission into the accepted set, but it may have undercounted at most two genuine migrations that were behaviour-perfect and zeroed at Stage I. Majority of three samples is load-bearing: without it, Stage I passes would fall from 340 to 305.
Stage III is not formal equivalence. The paper is explicit: six verifiers finding nothing is stronger evidence than one, not a proof. As models improve, the same 20 tasks will be scored more strictly. Drop the two opus verifiers and acceptances would rise from 28 to 46. Those two break 55.7% and 53.4% of their rounds; the other four sit between 21.6% and 26.1%. Stage II's all-or-nothing rule is also harsh: missing 1 check and missing 1,000 both score 0. That matches a drop-in replacement contract and also flattens near-misses to zero.