Private evals: 10-20 deeply understood cases beat statistical significance
isidoremiller · x · 2026-11-06
Responding to eval discussions: private evals will also inevitably be flawed and misleading, but the fix is to build a private eval set of just 10-20 cases and understand them maniacally at the atomic level, abandoning statistical significance. Public benchmarks must be 'big' to yield reliable comparative signal, which is why they're trash; a private eval is about your taste and judgment, so make it small enough that you never look at aggregate scores — just read the transcripts.
Related event: Hamel Husain: Private Evals Should Test Real Systems, 10-20 Cases Suffice(3 posts)→
More from coding & agent
- PHOS: A Raspberry Pi Robot That Gives Claude Desktop Sensors and Eyes via MCP — jfrankie · 2026-11-06
- Humans& multi-agent Ampersand matches frontier coding models at 70% lower cost — gharik · 2026-11-06
- Who Actually Runs Multiple AI Agents Daily? Reddit Weighs In — FulcraDynamics · 2026-11-06
- Memorilla: Memory System Cuts Tokens 4000x, 27x Faster Than Full Context — ShirleyYXWu · 2026-11-06
- Memorilla Compresses LLM Memory Into 16 Embeddings, Beats RAG Pareto Frontier — ShirleyYXWu · 2026-11-06
- Running parallel Claude Code agents on separate VMs, letting them manage CI/CD — kashifmanzoor · 2026-11-06