From-Scratch Model Trained with $1000 Reaches 11% on SWE-bench
sanmikoyejo · x · 2026-08-14
A new study from the Max Planck Institute and Stanford explores the limits of 'speedrunning' SWE-bench on a shoestring budget.
- Setup: Built upon Andrej Karpathy's nanochat, the model is trained from scratch on Qwen Coder agentic trajectories, with SWE-bench codebases strictly held out.
- Surprising Results: A mere $10 in compute yields a non-zero solve rate (0.53%). For $60 (approx. 12 B200-hours), the model achieves 5.0% pass@1, matching the SOTA at the benchmark's release (Claude 2).
- Scaling: Solve rate improves log-linearly with compute, reaching 11.0% at the $1000 mark—landing between Claude 3 Haiku and Claude 3 Opus.
The authors caution that climbing the benchmark without gains in underlying knowledge or math capabilities measures something narrower than general capability.
Related event: Researchers Train SWE-bench Model for $60, Rivaling Claude 2(2 posts)→
More from coding & agent
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- Claude Code and Codex now autocomplete the user's next action — tokenbender · 2026-10-16
- Hermes Agent ships official Reverse Engineer Anything plugin with one-command setup — Teknium · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16
- Cowork-Gym: Alibaba Synthesizes 64.9K Agent Work Environments, Boosting Qwen by 8.7 Points — Anni Zou · 2026-10-16
- MMSearch-OPD: Joint RL and On-Policy Distillation for Long-Horizon Multimodal Search Agents — PekingUniversity · 2026-10-16