How to measure the human cost of agent work: review time, rework and cost per task
bibryam · x · 2026-10-26
Michael Cutler proposes a collection protocol to replace assumed oversight time: measure the human effort and agent spend per accepted task at an agreed quality bar.
Key points: define the task class and eligibility rules upfront; register every task, not just shareable runs; use the business task as the unit (a rejected patch plus two revisions = one task, three attempts, no cost reset on new chats); log each attempt's disposition—accepted, rejected, or unreviewed—with reasons and evidence. He notes it's a proposed method, not a completed experiment.
More from coding & agent
- Non-CS Beginners Are Shipping Code Through CI/CD Pipelines with AI Help — curious_vii · 2026-10-26
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26
- Dev testing one skill across Opus 5.5 and Sol 6.1 finds instruction following 'all over the map' — lucasmeijer · 2026-10-26
- Hugging Face uses small open models to reverse engineer a game binary byte-for-byte — yb2698 · 2026-10-26
- ContractSim: first framework evaluating LLM negotiation over contingent natural-language contracts — xuanalogue · 2026-10-26
- OpenAI computer-use agents near superhuman speed, say the team building them — a16z · 2026-10-26