SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes
Silin Chen, Zihan Zhang, Xiaodong Gu, Jiale Huang, Yuling Shi, Han Li, Zhao Shan, Chengcheng Wan, Zhensu Sun, Haibing Guan
cs.SE
2026-10-15
Qwen3.5-9B fine-tuned on 13,407 reversed fixes gets a 61.2% realism vote, pushes agent Pass@1 down to 16.81%, and stays 88.27% unique at 200 samples per file.
A usable repository-level SWE task needs a real codebase, a behavioral bug that tests can observe, an issue that reads like a user report, and an environment that actually runs. GitHub-issue benchmarks grow only as fast as issue trackers, and environment setup tightens the bottleneck further. Collecting more real issues does not scale the test set.
SWE-smith prompts a model to edit or rewrite code, SWE-Synth rewrites components and filters with tests, and BugPilot keeps regressions that show up while an agent adds a feature. Those bugs go beyond hand-written mutation rules. The generator was never trained on how correct code gets broken, so repeated draws fall back on bug shapes the pretrained model already knows.
A historical fix records how a bug came about. Reverse the patch and repair supervision becomes bug-induction supervision grounded in real development. SWE-at-Scale samples that distribution on new code.
Training data comes from SWE-Fixer, SWE-rebench, SWE-bench-Live, and SWEGym, more than 134k issue-fix pairs. The developer's golden patch is applied to rebuild the fixed file, and a unified diff from that file back to the buggy version is the training target. Both directions have to apply cleanly in a round trip.
Non-code edits, formatting-only diffs, incomplete patches, and sequences over 64K tokens are dropped, as is every repository that appears in SWE-bench Verified. Balanced sampling plus a manual check leaves 13,407 transformations from 688 repositories. Logic and Interface are the most common categories because they already dominate real fixes.
Qwen3.5-9B reads the full correct file with line numbers and is not told where to edit, so it learns the site and the change together. LoRA fine-tuning runs for one epoch with a 64K context on a single RTX PRO 6000.
A candidate is kept only if it breaks at least one previously passing regression test (Pass-to-Fail, P2F). Another LLM then writes a GitHub-style issue.
Every method bugs the same file and the same correct state as each SWE-bench Verified instance, at most five times: temperature 0, then 1.0, stopping at the first P2F. SWE-smith may edit only the function that holds the original hunk. Solving uses mini-SWE-agent at temperature 0 with a 250-step cap. The model that wrote the issue does not solve it. Numbers are means of three runs.
Usability is whether five tries yield a non-trivial P2F, excluding edits that merely insert pass, TODO, or return None. Realism is the human preference rate: three engineers with at least three years of experience, majority vote, on which issue-patch pair in the same context looks more like ordinary development. Lower Pass@1 means a harder task.
| Method | Usability | HPR | Pass@1 |
| SWE-at-Scale | 72.6% | 55.9% | 31.68%-48.48% |
| SWE-at-Scale+ | 94.0% | 61.2% | 16.81%-37.02% |
| Best LLM-Rewrite | 72.2% | 51.8% | higher than both reverse-fix variants |
| FeatAdd (DS-V4-Flash) | described as much lower | realism stays high | 37.93%-62.07% |
| SWE-bench Verified | real issues | paired reference | 69.60%-97.00% |
| DeepSWE | expert-written | no matching vote | 24.30%-63.40% |
Across DeepSeek-V4-Flash, GPT-5.4-mini, GLM-5.3-Flash, and Qwen3.8-Flash, SWE-at-Scale scores 48.48%, 31.68%, 42.15%, and 44.44%. SWE-at-Scale+ scores 33.19%, 16.81%, 35.32%, and 37.02%. Scale+ is the lowest in every column, below Verified and the prompting baselines, and it sits in the DeepSWE difficulty band.
LLM-Modify, which asks a model to inject a bug at a chosen site, lands at the bottom of the preference ranking because the edits look like local mutations. Rewrites look closer to an accidental bad implementation, and the best of them still reaches only 51.8% HPR.
Mean bug-type entropy (MBTE) is the macro-average, over files, of bug-type entropy among usable candidates from that file. With the training count fixed at 2,000 and the repository count raised from 5 to 100, MBTE rises while usability stays roughly flat. On 50 files, MBTE is 0.527 at temperature 0 (69.0% usable), peaks at 1.173 at 0.6, and falls to 0.714 at 1.0 (23.4% usable).
At temperature 0.6, raising the budget from 5 to 200 samples per file lifts MBTE from 0.583 bits to 2.063. Usability levels off near 47% past 50 samples. Of 4,749 usable patches, 4,192 (88.27%) remain unique after normalized-diff dedup. LLM-Modify under the same budget reaches 1.22 bits, 44.1% usability, and 61.74% unique patches. Scale and Scale+ change 18.793 and 18.536 lines on average, against 14.334 for Verified fixes. Direct patch synthesis costs 17.45k tokens per instance, against 34.92k-62.82k for LLM-Rewrite and 2.29M for FeatAdd.
Environments cost more than bugs. A tested repository can be reused, and one file can supply many distinct failures: after 200 draws, 88.27% of usable diffs are still unique. A 9B model, one RTX PRO 6000, and one LoRA epoch turn 13,407 historical fixes into that generator, at a per-instance cost more than two orders of magnitude under FeatAdd.
Teams that already have tested Python repositories can expand an eval set this way. Against prompt-based injection, the gains in diversity and difficulty are concrete. Against human-written benchmarks, the tasks remain synthetic. Scale+ is harder and more usable, and it adds a teacher model plus an executor. The direct-diff variant is the one to reproduce. The paper does not test whether agents trained on these tasks improve on real issues.
The study covers Python only. There is no transfer experiment for other languages. Bugs that no test covers are discarded, so thin suites produce few tasks.
An HPR of 61.2% means raters picked the synthetic sample more often than the paired real issue. There are three raters, and the paper reports neither the number of pairs nor agreement. The vote can favor a clean edit that breaks a test on cue. That falls short of the synthetic bugs being more realistic than real issues.
Pass@1 mixes bug difficulty with the quality of an LLM-written issue. A vague report lowers the solve rate by itself. How much of Scale+'s drop comes from a deeper bug is not separated out. The 94% usability figure means at least one success in five tries, with early stop. At temperature 0.6 the per-sample yield is about 47%. Those are different denominators.
Scale+ patterns are authored by DeepSeek-V4-Flash. The 9B model imitates them, and an executor edits the code. There is no control that freezes the executor and swaps in patterns from an untrained model, so the student's share is unclear.
Moving from 14.3 to 18.8 changed lines is not the same as a more complex bug. The 88.27% figure is diff uniqueness, not behavioral uniqueness. MBTE is still rising at 200 samples, but it has slowed after 100. Synthesis is also anchored to Verified files. Baselines are confined to the function that held the original fix, while Scale picks its own site, so the difficulty gap is not only a richer bug distribution. The solver scaffold is mini-SWE-agent alone.