Ofir Press lays out 3 rules for post-human-level benchmarks: tasks humans need years or decades to solve
OfirPress · x · 2026-10-13
Benchmark researcher Ofir Press argues that in the human-level AI era, many human tasks are now too easy to translate directly into benchmarks. In a new post he offers three guidelines for building benchmarks in this new era:
- Use tasks that would take teams of humans years or decades to solve.
- Find real-world tasks real humans actually had to solve — don't invent them or have humans fabricate them for the model.
- Choose tasks that can be scalably crawled from the web.
Related event: Ofir Press calls for new benchmark design as human baseline era ends(2 posts)→
More from Research
- OncologyBench: Latch's 25-Task Multiagent Cancer Benchmark Tops Out at 40% Pass — kenbwork · 2026-10-14
- Google's Gemini 4 Argon for robotics: simulation training lifts task success from 56% to 75% — DynamicWebPaige · 2026-10-14
- Alex Wa's deep dive distills training methodologies of 7 open-weight frontier models, from MoE to data recipes — goyal__pramod · 2026-10-14
- OccludeBench: VLMs are 20% worse at reasoning from occluded physical cues — liuzhuang1234 · 2026-10-14
- OpenMined funds AI ensembling research: $2k stipend plus $5k compute per researcher — iamtrask · 2026-10-14
- Chantless research trace: six months from arithmetic-MLP experiments to full-model specialists — tokenbender · 2026-10-14