Auditing 1,228 Human Interventions: 91% Weren't Decisions, So They Stopped Trusting Agent "Done"
kraboo_team · reddit · 2026-09-28
A team running 25 AI agent workspaces (PM → Dev → Tester → Critic → Release) audited 8 weeks of human interventions and found only 9% were real decisions; 59% of tasks were closed by hand despite agents claiming completion, and 8-11 per 100 tasks were false "done" claims (e.g., "30 files pushed" with nothing on the remote).
Their fix: keep agents as-is but add a small MCP-backed verification service:
- Per-task-type "work contracts" listing what must be true (links resolve, release package on remote branch, etc.)
- Checks run by the service on pushed branches, invisible and uneditable by agents, some hidden
- Three states: PASS / FAIL / UNVERIFIED — anything unproven is never faked as a pass
- Only the service can set "Done"
They're starting in shadow mode and asking the community about measuring verifier miss rates, subjective checks without LLM judges, and continuous re-verification after "done".
More from coding & agent
- Vivek Haldar on 'meat proxies': don't forward AI output unread, use agents with rigor — vivekhaldar · 2026-09-28
- Qwen 3.8 27B Fixes Bugs in 10 Minutes; Flash Next Takes 2.5 Hours — poofph · 2026-09-28
- Free Public MCP Server for Canadian Privacy Law Data, No Auth Needed — masiha97 · 2026-09-28
- Muse's phone-call feature sparks debate: agentic commerce needs agent-to-agent interaction — bansalg_ · 2026-09-28
- The Hardest AI Agent Failure: A Cheerful Reply to an Angry Customer — BinVio · 2026-09-28
- Developers file detailed issue on Copilot Chat UX and reliability gaps in VS Code — DanWahlin · 2026-09-28