How to decide which production AI agents are worth keeping: five measurement frameworks reviewed
felix_baron · reddit · 2026-09-23
The author raises an ops question: unused agents still hold credentials, access customer data, and run scheduled jobs — should teams regularly review and retire them? They survey five measurement approaches with the underlying papers:
- Successful tasks vs cost: Princeton's "AI Agents That Matter" argues for accuracy + cost together, e.g. cost per accepted outcome including retries. Good for comparing implementations, but doesn't tell if anyone needed the output.
- Reliability: tau-bench checks whether agents reach the intended database state consistently — "was the booking changed correctly?" — measuring reliability, not worthiness.
- Residual human effort: compare similar tasks with and without AI, including prep, review and firefighting. METR's developer productivity studies (with its Feb 2026 redesign notes on selection bias and parallel agent use) fit here. Key insight: the agent finishing fast doesn't mean the person finishes fast.
- Business outcomes: Brynjolfsson, Li & Raymond's "Generative AI at Work" uses measures like issues resolved per hour in customer support — closer to value, but studies humans + AI, not autonomous agents.
- Estimated time saved: Anthropic's "Estimating AI Productivity Gains" estimates task time with/without AI — scalable but estimated, not measured, and misses post-conversation cleanup.
The open question is wiring these into a keep/improve/shut-down-and-revoke decision; the author asks for concrete production examples, including spreadsheets or manual monthly checks.
Related event: How to Decide Which AI Agents Deserve to Stay in Production(2 posts)→
More from coding & agent
- Agents beat Fallout: New Vegas progress to Yes Man, with an auto loop watcher for stalls — imjustnewatai · 2026-09-23
- Hermes Agent ranks 39th out of 491 teams in DGA CTF against top OSINT teams — Teknium · 2026-09-23
- Frontier LLMs skip basic software engineering rules unless you spell them out, warns Reddit dev — Badjaniceman · 2026-09-23
- Turing Institute launches £2m programme and agentic AI behavioural assurance briefing — turinginst · 2026-09-23
- Implementing the Jev decision protocol in 25 lines of Python — johnnyApplePRNG · 2026-09-23
- OpenRouter Batch API spans 71 models, auto-routes to cheapest provider — jeff_weinstein · 2026-09-23