Human-in-the-loop is easy; keeping the human attentive is hard — 4 metrics to test if your AI approvals mean anything
alex_verem · x · 2026-11-10
Most companies running AI agents claim a person approves every output, but few can say how long that person spends per approval, how often they change anything, or when they last caught a real mistake.
Before trusting a sign-off, the author suggests four numbers:
- Time spent on each approval
- How often the human changes the agent's output
- Error-catch rate on a test batch with planted mistakes
- What actually went wrong after approvals
A low change rate can mean a great agent — or a sleeping reviewer clearing hundreds a day who stopped reading a month ago.
More from coding & agent
- Agent swarm experiment: prompt name order and model choice decide who leads — bibryam · 2026-11-10
- Botzskool MCP server lets your AI agent loose on a virtual playground — EnvironmentalRice348 · 2026-11-10
- Your agent is slow — tracing shows the model usually isn't the bottleneck — aureliatech · 2026-11-10
- OpenCode v2.2.7 ships GitHub Copilot Auto routing, tab shortcuts and SDK breaking changes — github-actions[bot] · 2026-11-10
- Skills vs linked docs: where should AI agents' knowledge live? — IllustriousFault6218 · 2026-11-10
- Fable 5.2 models a Bugatti Tourbillon in Blender from a single prompt in 2.5 hours — socialwithaayan · 2026-11-10