If your evals pass 100% of the time, they've stopped telling you anything
HamelHusain · x · 2026-10-21
Robertta relays Hamel Husain's point: if your evals pass 100% of the time, they've stopped telling you anything.
He calls this saturation — you've already conquered the test, so there's nothing left to learn from it. The fix is to keep climbing: give yourself progressively harder problems so your evals retain diagnostic value.
More from coding & agent
- Developing Taste: Emil Kowalski on why judgment beats tools in the AI era — rseroter · 2026-10-21
- MCP in action: Claude Opus hands off asset library videos to Runneth, launching Friday — jacob_posel · 2026-10-21
- Mastercard taps Coinbase's x402-based infrastructure to let AI agents pay for digital services — MurrLincoln · 2026-10-21
- RoboARC shows robots get no free retries—unlike coding agents that can just rerun tests — ManlingLi_ · 2026-10-21
- Anthropic open-sources 11 Claude plugins for sales, legal, finance and more knowledge work — mhdfaran · 2026-10-21
- Pixlet display turns Claude agent status into a physical dashboard — whurley · 2026-10-21