Half of PM job listings now demand evals skills: Ramp 35%→83%, Shopify 2.2x faster at 68% cheaper
lennysan · x · 2026-09-23
Lenny Rachitsky notes evals keep coming up: nearly half of 25 top PM job openings he shared ask for eval-writing experience. Companies report concrete returns:
- Ramp lifted automatic receipt-collection accuracy from 35% to 83%
- Shopify shipped an AI workflow builder 2.2x faster and 68% cheaper than its frontier-model setup
- Harvey rebuilt its AI contract reviewer, nearly doubling internal quality scores
- Cursor tuned Auto Balance routing for higher satisfaction at 41% lower cost
He followed up with evals experts Hamel Husain and others — the takeaway: evals are now a core AI product-engineering skill, not optional.
Related event: Half of PM Job Listings Now Demand Evals Experience(2 posts)→
More from coding & agent
- qBotica: 3 founders to ~200 staff and 335% revenue growth by turning RPA into agentic AI software — rschmelzer · 2026-09-23
- A one-line prompt trick: 'Show me irresistible proof that you have fixed it' — jobergum · 2026-09-23
- Stripe's Harbor: AI-assisted prototyping tool produced 12,000 prototypes since May — dl_weekly · 2026-09-23
- Zvi on Opus 5.5 System Card: Failure Modes Are 'Largely Avoidable' via Prompting and Harness — TheZvi · 2026-09-23
- GPT-6 Sol and Luna spotted in OpenAI Codex, release imminent — mark_k · 2026-09-23
- AstroForge to launch spacecraft that gets zero commands from Earth next year — julianweisser · 2026-09-23