dair-ai founder: stop benchmaxxing, the next big opportunity in AI is building real evals
omarsar0 · x · 2026-11-01
dair-ai founder Elvis Omari argues that knowing how to properly measure what you want to improve—whether a model, a complex codebase, or a business function—is the core skill most practitioners lack. That's why today's AI excels mainly at verifiable problems; the open-ended world demands a fundamentally different approach to evaluation and improvement.
He criticizes short-term benchmaxxing and calls for a return to "hardcore science and evals," framing open-ended evaluation as a huge opportunity space. dair-ai is doing research and building evals here, and he invites collaborators and perspectives.
Related event: DAIR.AI founder: Without evals, no real improvement is possible(4 posts)→
More from AGI Musings
- Weekly: Anthropic's S1 filing, labs claim models refute Exponential Time Hypothesis — njyx · 2026-11-01
- Pedro Domingos revives Moravec's paradox: AI cracks Navier-Stokes but can't do accounting — pmddomingos · 2026-11-01
- Neuroscientist weighs in: AI consciousness unsettled, non-biological systems still in play — davidmanheim · 2026-11-01
- Switzerland is the only country with both very high AI adoption and spend — DesireeCachette · 2026-11-01
- Reddit debate: cloud LLMs change rules on a whim — only local models are reliable — Substantial_Swan_144 · 2026-11-01
- "2 years to escape the underclass" was AI industry fear marketing to boost valuations — Signalman23 · 2026-11-01