Deciding which AI agents earn their keep in production: five measurement approaches compared
felix_baron · reddit · 2026-09-23
The author raises a production pain point: unused agents still hold credentials, customer data access, and scheduled jobs—how do you decide to keep, fix, or retire them and revoke access? Five measurement approaches and their shortcomings:
- Successful tasks × cost: Princeton's AI Agents That Matter argues for evaluating accuracy and cost together (e.g., cost per accepted outcome incl. retries), but can't tell whether anyone needed the task.
- Reliability: tau-bench checks whether an agent reaches the intended database state consistently, but benchmark reliability ≠ business value.
- Leftover human work: Compare prep/review/correction/takeover time with vs. without AI; METR's controlled developer-productivity studies (Feb 2026 update) note selection bias and parallel agent use complicate measurement—an agent finishing fast doesn't mean the person does.
- Real operational outcomes: Generative AI at Work uses outcomes like issues resolved per hour, closer to business reality but studies human-AI collaboration, not autonomous agents.
- Estimated time saved from usage data: Anthropic's Estimating AI Productivity Gains estimates task time with/without AI from conversations—scalable, but estimates aren't observed savings.
The open question: how to connect these measurements to a concrete keep/fix/retire decision, and how to count human review and cleanup without making measurement another job.
Related event: How to Decide Which AI Agents Deserve to Stay in Production(2 posts)→
More from coding & agent
- Turing Institute launches £2m programme and agentic AI behavioural assurance briefing — turinginst · 2026-09-23
- OpenRouter Batch API spans 71 models, auto-routes to cheapest provider — jeff_weinstein · 2026-09-23
- Cap sub-agents with fixed turn and time limits to force real prioritization — RexDouglass · 2026-09-23
- StackBlitz open-sources Bolt Slides: agents build presentations that are live web apps — tom_doerr · 2026-09-23
- User runs fully automated SEO with an AI agent: one new page daily, shipped via PR — ayushtweetshere · 2026-09-23
- Grok Build tests Remote Control to run your PC from phone, hinting at a Grok desktop app — testingcatalog · 2026-09-23