Private evals vs third-party benchmarks: Hamel Husain sparks debate on model selection
mert_gulsun · x · 2026-11-09
Hamel Husain argues developers rely too heavily on third parties for model selection and should run product-specific private evals on their own tasks. mertgulsun counters that the two are complementary: keep your own evals, but independent institutions without conflicts of interest still add value for costly, complex evaluations with published methodologies.
More from coding & agent
- Revolut runs coding agents for 2,000 engineers at 1/10th cost with open models — zainhas · 2026-11-09
- Future CEO of agentic orgs will be a system designer, not a McKinsey alum — GregCook2011 · 2026-11-09
- When humans become the bottleneck, lift the loop: System 1 steers, agents deliver — dotey · 2026-11-09
- Run Claude Code on Your iPhone: Rex + Tailscale Setup Guide — leslysandra · 2026-11-09
- Ex-OpenAI RLHF lead Diogo's north star: make intelligence as boring as databases — schwentker · 2026-11-09
- Grok Ghost demo shows autonomous computer-use agent driven by screen OCR — Aiden_Tech_Ai · 2026-11-09