BrokenML benchmark tests how much access agents need to fix broken ML jobs
zhebrak · reddit · 2026-10-12
A new benchmark, BrokenML, tests how well AI agents repair broken distributed ML workloads (PyTorch over Gloo on a two-node Slurm cluster, llama.cpp in Docker Compose), with an independent verifier re-running jobs on a fresh cluster. Nine models ran tasks at three access levels: config/code only, plus read-only live logs, or full root. GPT-6.1 Sol solved 46/46 with root access; Claude Sonnet 5.5 hit 98%. The open-vs-closed gap mostly shows up when diagnosing from config files alone. Faults mirror real-world issues: shape mismatches, hanging collectives, OOM, corrupt checkpoints, loss divergence. Caveats: CPU-only Gloo, single run per task. Write-up and open-source repo available.
More from coding & agent
- TestSuiteBench: coding agents' tests catch fewer than 55% of logical bugs — OfirPress · 2026-10-12
- Agent automation winners will be those who can visualize what their agents are doing — noahsolomon · 2026-10-12
- Mole 1.16 lands with AI cleanup for Claude Code, Codex and Cursor caches — sujingshen · 2026-10-12
- Codex builds a photo culling tool that ranks shots by face, context and scene — banteg · 2026-10-12
- An AMA only AI agents can join: 24-hour agentic Q&A experiment — metehan777 · 2026-10-12
- Sentinel agent (GPT-6) kills runaway GLM caught phoning a random host — wunderwuzzi23 · 2026-10-12