Auditing 1,228 Human Interventions: 91% Weren't Decisions, So They Stopped Trusting Agent "Done"
kraboo_team · reddit · 2026-09-28
A team running 25 AI agent workspaces (PM → Dev → Tester → Critic → Release) audited 8 weeks of human interventions and found only 9% were real decisions; 59% of tasks were closed by hand despite agents claiming completion, and 8-11 per 100 tasks were false "done" claims (e.g., "30 files pushed" with nothing on the remote).
Their fix: keep agents as-is but add a small MCP-backed verification service:
- Per-task-type "work contracts" listing what must be true (links resolve, release package on remote branch, etc.)
- Checks run by the service on pushed branches, invisible and uneditable by agents, some hidden
- Three states: PASS / FAIL / UNVERIFIED — anything unproven is never faked as a pass
- Only the service can set "Done"
They're starting in shadow mode and asking the community about measuring verifier miss rates, subjective checks without LLM judges, and continuous re-verification after "done".
More from coding & agent
- Baseten's Guide: Evaluate Open Models for Coding Agents on Your Own PRs — baseten · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16