SWE-sweep: new benchmark tasks agents with finding bugs before users hit them
klieret · reddit · 2026-10-03
Researchers from Meta, Stanford, Harvard and UW released SWE-sweep, a benchmark testing whether models can find and fix bugs before any user encounters them.
- Setup: hand an agent a full codebase and ask it to find & fix as many bugs as possible; scoring is based on a hidden set of known real-world bugs in the repos.
- Rigorous filtering ensures bugs are actually discoverable and fixable from reading the code alone.
- Early observations: scores are much lower than expected; fixing all of numpy is legitimately superhuman, but models also underperform on many small repos. Luna xhigh currently leads on cost efficiency.
- Fully open source (MIT), with more open-weight models to be added to the leaderboard soon.
Related event: Meta's SWE-sweep: Best Models Fix Only 4.7% of Hidden Bugs(6 posts)→
More from coding & agent
- Open-source Switch Buddy puts Claude Code, Codex and other agents in shared rooms — anthara_ai · 2026-11-19
- Stanford CS329Z Lecture 5 compares DSPy vs. LangGraph agent framework approaches — stanfordnlp · 2026-11-19
- Mistral Large 4 coding test: 2 hours of agentic work burned just €0.82 of quota — qtnx_ · 2026-11-19
- Railway breach investigator: LLM attacker's infrastructure still targeting other AI companies — cyb3rops · 2026-11-19
- Gradio and HF launch hackathon: reproduce NeurIPS 2026 papers with AI agents — Gradio · 2026-11-19
- NVIDIA open-sources SkillEvaluator to vet AI agent skills before deployment — anthara_ai · 2026-11-19