Google's BuggyMLE injects bugs into ML repos, dropping top model solve rate from 95% to 9%
omarsar0 · x · 2026-11-23
Google Research and DeepMind released BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection, tackling the rapid saturation of repository-level coding benchmarks by deliberately injecting controlled bugs into ML engineering tasks.
Key points:
- Introduces the Code Complexity Index (CCI), built from ten structural repo features (scale, dependency depth, patch size), which predicts average solve rates across eight coding benchmarks with R²=0.951; patch size and complexity are the strongest difficulty predictors.
- Agents inject controlled mutations into 222 working ML repos, synthesizing 6,600+ bug-injected variants and 260K lines of injected code to create BuggyMLEBench-Lite, the first difficulty-graded repository-level variant of MLE-Bench Lite.
- Across 14 evaluated models, the top solve rate falls from 95% on the easiest tier to 9% on the hardest.
- A Qwen3.5-9B fine-tuned on the bug curriculum rivals frontier models on the benchmark and lifts SWE-bench Lite from 24% to 33% and LiveCodeBench v6 pass@1 from 69% to 71%.
More from Models
- OpenAI chart reveals unreleased GPT-6 Astra models; Domingos questions billions for <0.3 bits — pmddomingos · 2026-11-23
- Replit agents speed up NumPy 18.9x — but only Claude's team scales — amasad · 2026-11-23
- Perplexity is a flawed cross-lab capability bound: tokenizer granularity favors OpenAI — davidmanheim · 2026-11-23
- Cartesia's H-Net ditches tokenization, learns dynamic byte grouping for up to 10x training efficiency — bycloud · 2026-11-23
- Gemini 4 Hydrogen at $1.50/M Input Tokens Scores 51, Beating Argon's 50 — dl_weekly · 2026-11-23
- Base log perplexity is an extremely strong predictor of Epoch and OpenAI ECI, researchers note — stochasticchasm · 2026-11-23