Advanced AI evals: most teams skip error discovery and measure the wrong things
lennysan · x · 2026-09-23
In a deep-dive Lenny's Newsletter post, Hamel Husain and Shreya Shankar distill work with 50+ AI companies into a guide for finding and fixing hidden AI failures. Their core claim: most teams jump straight to writing metrics and end up measuring the wrong things — the skipped step is error discovery.
Case data: Ramp lifted automatic receipt-collection accuracy from 35% to 83%; Shopify shipped an AI workflow builder 2.2x faster and 68% cheaper than the frontier-model setup it replaced; Harvey nearly doubled internal quality on its rebuilt AI contract reviewer; Cursor tuned Auto Balance routing for higher satisfaction at 41% lower cost.
The post details which steps can and cannot be automated, plus a free plugin that lets a coding agent do most of the heavy lifting, alongside a 30-minute read and ready-to-use evals skills. Nearly half of recent AI PM job openings ask for evals experience.
Related event: Husain and Shankar Publish Deep Guide on AI Evals(4 posts)→
More from coding & agent
- Pigment Agents now reads and posts in Slack, inheriting user permissions — mattturck · 2026-11-12
- Map of the agent economy: 50+ startups across payments, spend control, identity, memory — venturetwins · 2026-11-12
- OpenCode launches Browse, an on-device browser agent with local logins — Saul_Loveman · 2026-11-12
- VentureTwins maps the B2A economy: dozens of agent infra and app companies in first series installment — venturetwins · 2026-11-12
- Google releases 76-page 'Agents Companion' whitepaper on agent engineering — mdancho84 · 2026-11-12
- Spinning up containers for 20-line agent tools is backwards: Safescript compiles provable, 50ms-deploy apps — uriwa · 2026-11-12