Bug injection turns ML repos into a difficulty dial: top model solve rate falls from 95% to 9%

BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu

cs.AI, cs.LG, cs.MA, cs.SE

2026-11-19

Controlled bug injection makes ML repos a difficulty-graded benchmark: top solve rate falls 95%→9%, and a fine-tuned 9B model beats Claude Opus 5.5 on the 10-bug tier.

What problem this solves

Benchmarks for machine learning engineering (MLE) agents share three flaws. They grade single-file scripts, so the agent never touches a real multi-file repository or an open shell. They saturate, with frontier models beating human reference solutions within months. And every task is hand-curated: each one needs a vetted dataset, an execution harness, and a human-verified outcome. The deeper gap is that nobody could quantify what makes a repository-level task hard, so synthetic task factories produce volume without a difficulty gradient.

A Google Research and DeepMind team (with UIUC and Georgia Tech co-authors) built both the measurement and the control.

Method

The first piece is the Code Complexity Index (CCI). Ten structural features per benchmark, grouped into repository scale (lines, files, directory depth), dependency depth (count, longest chain), and patch scope (files changed, hunks, lines added or removed). Features are log-transformed, z-scored, averaged within groups, and fitted with a four-parameter logistic regression to predict a benchmark's average solve rate. No model information enters the index. Across eight public coding benchmarks it explains 95.1% of the variance (leave-one-benchmark-out MAE = 0.048). Patch scope carries the largest coefficient (-0.99, versus +0.64 for repo scale and -0.34 for dependencies) and alone explains 75.6%. Adding model identity, as fixed effects or an external intelligence index, makes out-of-sample predictions worse: benchmarks sit in structural floor-or-ceiling regimes, and capability only separates models in the middle.

The second piece is BuggyMLE, a four-stage pipeline:

The injected defects read like routine engineering changes: calibration returning 1/T instead of T, SpecAugment masking 45% of both axes, label smoothing set to 0.05, each with a plausible justifying comment. Total scale: 222 source repositories, 6,674 execution-verified variants, about 260K injected lines.

Results

BuggyMLEBench-Lite is the repository-level variant of MLE-Bench Lite: 22 multi-file repos across four tiers (1/5/10/50 bugs), 88 tasks, with solve rates falling monotonically for 13 of 14 models.

SettingResult
Best on 1 Bug tier (Claude Opus 5.5)95.5%
Best on 50 Bugs tier9.1%
Untuned Qwen3.5-9B, 1 Bug tier13.6%
Full-curriculum 9B, 1 Bug tier72.7% (SFT-only: 50.0%)
10 Bugs tier, curriculum 9B18.2%, above Opus 5.5, behind only Sonnet 5.5
50 Bugs tieronly Opus 5.5 and Gemini 3.8 Flash solve any task fully

Training uses 200 extra repos in three bands (1–5, 10–12, 30–34 hunks): SFT on 2,724 verified teacher trajectories plus 411 RFT rollouts, then GRPO. The ablations carry the message. Easy Only scores 0% on both the 5 and 10 Bugs tiers. Hard Only drops to 22.7% on 1 Bug despite resolving more individual bugs (59.1% versus 50.0%). Only the full curriculum wins on both ends. Transfer: SWE-bench Lite resolved rate rises from 24.3% to 33.0% and LiveCodeBench v6 pass@1 from 69.3% to 71.3%, while both single-difficulty curricula land below the untuned base. On what makes tasks hard, silent bugs (validation metric untouched, test predictions corrupted; one off-by-one keeps validation at 0.953 while test falls to 0.017) show the lowest resolution rates across all models, as do bugs rated 4–5 for realism.

Why it matters

Two reusable artifacts. CCI is a cheap difficulty gauge: ten structural features predict solve rates, so benchmark builders can check how much headroom a synthetic suite has left. BuggyMLE turns difficulty into a dial, same repo, same dataset, turn the hunk budget. For anyone training coding agents that is a ready-made curriculum factory with dense verifiable rewards. The 9B model overtaking frontier models on the 10-bug tier also shows this capability can be built with data rather than parameter count.

Limitations

The authors list three: CCI is validated on eight human-curated benchmarks only; it is purely structural, blind to semantic complexity, and weaker on standalone-program benchmarks like FrontierCS Algo; transfer to from-scratch ML generation is mixed. Two more from reading closely. The 50 Bugs tier first expands each repository to roughly twice its size before injecting 23–60 bugs, so part of the difficulty comes from the larger codebase, not the bug count. Per-bug verification relies on an LLM judge: Opus 5.5 recovers the target metric on most 10-bug tasks yet is marked down for leaving bugs in place, so rankings depend on the grading policy. The benchmark also rests on only 22 underlying repositories, so tasks are correlated.

Terms

Source

What people are saying

Related papers

All paper explainers