Hamel Husain: private evals must measure your real system, not abstract proxies
HamelHusain · x · 2026-11-06
Replying in a discussion, Hamel Husain stresses that 'private evals' aren't about privacy but about your specific tasks: stop building another private benchmark that abstractly represents what you care about, and measure your real system instead. He adds that private benchmarks usually suck too — what he means is high-quality tests on your real system that are constantly maintained.
Related event: Hamel Husain: Private Evals Should Test Real Systems, 10-20 Cases Suffice(3 posts)→
More from coding & agent
- Yukon syncs tables near-real-time at 1M events/sec, built so agents can query what changed — testingcatalog · 2026-11-06
- How one creator turned X posts into 40,000 email subscribers with a Go/No-Go AI agent — VibeMarketer_ · 2026-11-06
- Three open-source models combine into a fully local, real-time voice-to-avatar assistant — shinjiw_at_cmu · 2026-11-06
- Garry Tan: Capy's coordinating subagents boosted his PR output ~13x — garrytan · 2026-11-06
- GitHub Copilot's agent stack: cross-repo projects, parallel sessions, desktop app and automations — 0xkarasy · 2026-11-06
- Stripe opens its data to agents: free queries through Jan 2027, SQL shown with every answer — emilygsands · 2026-11-06