Hamel Husain: build private evals with 10-20 cases, understand them atomically
HamelHusain · x · 2026-11-06
In a discussion with isidoremiller about whether private evals inevitably suck, Hamel Husain shares a concrete method: yes, they can — so build a private eval set with just 10 or 20 cases and maniacally understand them at the atomic level, forgetting statistical significance.
His take: public benchmarks are trash because they must be 'big' to achieve statistical reliability, but that's orthogonal to the insight engineering decisions actually need — a small, deeply understood test set on your real system beats a large abstract benchmark.
Related event: Hamel Husain: Private Evals Should Test Real Systems, 10-20 Cases Suffice(3 posts)→
More from coding & agent
- Ripple-65B preview lands on Together AI: 80.5 on Terminal-Bench 2.1, 1M context — togethercompute · 2026-11-06
- Garry Tan: Capy's coordinating subagents boosted his PR output ~13x — garrytan · 2026-11-06
- GitHub Copilot's agent stack: cross-repo projects, parallel sessions, desktop app and automations — 0xkarasy · 2026-11-06
- Stripe opens its data to agents: free queries through Jan 2027, SQL shown with every answer — emilygsands · 2026-11-06
- Stripe opens MCP access: query your payments data from Claude or Cursor in plain English — emilygsands · 2026-11-06
- Stripe opens data to agents via MCP: plain-English queries, dashboards and board decks — emilygsands · 2026-11-06