Rox replaces manual evals with Devin, lifting coverage to 72% of real traffic
DevinAI · x · 2026-10-16
- AI startup Rox says nobody on its team writes evals anymore: Devin now QAs its agent weekly on real traffic.
- Workflow: Devin reads real conversations with Rox's agent, rebuilds each failure in a full copy of the product, and turns it into an eval — humans just review the PR.
- The suite already covers 72% of real product usage vs. 43–57% for the hand-written evals it replaced, and coverage keeps growing week over week.
- Built with Cognition and Braintrust, it's a notable case of automated eval engineering for agents.
More from coding & agent
- MIT to host ScienceClaw agent hackathon on multi-agent scientific discovery, apps close Oct 16 — ProfBuehlerMIT · 2026-10-16
- AI review agents catch different bugs than humans — together they find 82.1% — rseroter · 2026-10-16
- Liquid AI's LFM2.5-24B-A2B tested locally: 2B active params for on-device agents — JosephJacks_ · 2026-10-16
- Four workflow tips for driving Gemini 4 Argon in Antigravity CLI — rseroter · 2026-10-16
- OpenCode 2.0 wins over Hacker News with auto-triaging GitHub App and opentunnel — ycombinator · 2026-10-16
- VOYGR launches PlaceCall on YC: an API that lets AI agents phone US businesses — ycombinator · 2026-10-16