SWE-bench Co-author Ofir Press: Good Benchmarks Should Start Near 0% Accuracy
OfirPress · x · 2026-11-21
On Kaggle's Evaluated podcast, SWE-bench co-author Ofir Press outlined common mistakes benchmark builders make:
- Starting too easy: if a benchmark launches at 50% accuracy, the objective is nearly solved. Best models should score around 0% on launch day. SWE-bench launched with top models solving only 2% of real GitHub bug tasks; for 3-4 months people thought it was too hard or impossible, until Cognition cited it in the Devin launch and competition took off. Today models solve most of it.
- Choosing tasks nobody cares about: pick natural tasks people actually want models to do well.
His design rule: tasks that are "easy to explain, easy to judge, but extremely hard to execute."
More from Models
- Models Afraid to Show Strategic-Level Thinking, Claims an RSI Overhang — repligate · 2026-11-21
- Inkling 2 Mini: 26B looped transformer runs agentic work fully offline on a laptop — simonguozirui · 2026-11-21
- Looping models could bring Opus-level vibe coding to small open models on your phone — scaling01 · 2026-11-21
- Two new benchmarks push AI evals past static testing: recursive self-improvement and KradleBench — typewriters · 2026-11-21
- KOL endorses Google AI Ultra: screenshot shows up to 5x Gemini limits vs Pro — kimmonismus · 2026-11-21
- Claude's claimed Riemann proof mocked: 250,000 pages of unreviewed Lean, author calls it 'slop' — GaryMarcus · 2026-11-21