Half of 25 PM Job Openings Ask for Evals Experience; Ramp, Shopify, Harvey, Cursor Share ROI Data
lennysan · x · 2026-09-22
Lenny Rachitsky notes evals are becoming a core PM skill: nearly half of 25 PM job openings he shared require evals experience.
Companies' reported returns on eval investment:
- Ramp: automatic receipt collection accuracy up from 35% to 83%
- Shopify: AI workflow builder 2.2x faster and 68% cheaper than the frontier-model setup it replaced
- Harvey: rebuilt its AI contract reviewer, nearly doubling internal quality score
- Cursor: tuned Auto Balance routing, much higher satisfaction at 41% lower cost
He also commissioned Hamel Husain and Shreya Shankar, drawing on work with 50+ AI companies, to write an advanced sequel to their popular guide on building eval systems.
More from coding & agent
- StepFun open-sources Step Code: 80.9% on Terminal-Bench 2.1 with fewer tokens — StepFun_ai · 2026-09-23
- X exec: bot detection will be an urgent business as agent swarms flood the web — nikitabier · 2026-09-23
- xAI Console Redesigned With Playground to Test Grok 4.7 and Full API Stack — XFreeze · 2026-09-23
- HelloSol: a local-first AI agent that closes your 'I'll get back to you' loops — SucceededMind · 2026-09-23
- Compound Engineering 3.25 adds one-command refresh for compounded learnings — kieranklaassen · 2026-09-23
- Firecrawl raises $75M Series B and launches Alexandria knowledge library for AI agents — devdigest · 2026-09-23