Most benchmarks are made by vendors chasing #1 — how to stress-test them yourself
VikParuchuri · x · 2026-11-20
Vik Paruchuri argues benchmarking is hard even for experts, and most benchmarks are built by vendors aiming to rank #1 rather than help you judge fairly. He shares a talk on how to stress-test benchmarks yourself, plus a quick checklist, and recommends setting up your own evals on your own data whenever possible.
Related event: Datalab's Vik Paruchuri Audits OCR Benchmarks and Introduces OmniParseBench(5 posts)→
More from Research
- Perception verbs alone identify fiction with 90% accuracy, decision tree shows — _akpiper · 2026-11-20
- Ai2 launches open-source Theorizer in Asta: AI proposes testable scientific theories — allen_ai · 2026-11-20
- AI2 launches Theorizer in Asta, an AI that proposes testable scientific theories from literature — allen_ai · 2026-11-20
- Cross-institutional survey asks whether mechanistic interpretability actually makes AI safer — sarahookr · 2026-11-20
- μSAM2 released: SOTA microscopy segmentation trained on 30M+ annotated cells — alxndrkalinin · 2026-11-20
- SnakeBench: frontier models still fail to plan more than 2-3 moves ahead — GregKamradt · 2026-11-20