From no benchmarks in the 1990s to no papers without them: a short history of ML evaluation
abursuc · x · 2026-09-16
A clip from the #ssad2026 talk: Kashyap Chitta traces the history of ML evaluation — in the 1990s almost no papers had benchmarks, while in the 2020s no paper ships without one. Establishing a common test framework culture was hard, but the resulting progress proved impressive.
Related event: SSAD 2026: Rethinking AV Benchmarks and the Case for Open Source(5 posts)→
More from Research
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02
- Karpathy's nanochat: the best ChatGPT $100 can buy, near 60k GitHub stars — karpathy · 2026-11-02
- After reading 46 resources on agent evals, one author explains why her 95% score was misleading — blaizedsouza · 2026-11-02