Ofir Press on benchmark-building: hard, verifiable, desirable — but 'cheap' is out
maxbittker · x · 2026-10-20
A slide shared by maxbittker came from a chat with benchmark researcher Ofir Press, who said the hardest part of his job is building benchmarks that are compelling, legible, and worth paying attention to. The slide lists the criteria for a good benchmark: Hard, Verifiable, Cheap, Desirable, and Interesting to other people. Press added in a reply that they've largely given up on 'Cheap' since many good ideas end up expensive, but strongly agrees with the rest.
More from Research
- OPAL: an LLM agent rewrites the optimizer's source code during training — RobertTLange · 2026-10-23
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23