Execution Success Isn't Verification: 60/60 Tasks Ran, 0/60 Were Correct
DMAE1133 · reddit · 2026-09-19
A benchmark developer hit a failure mode agents builders underestimate: an ablation completed 60/60 tasks with correct topology activation but produced 0/60 correct answers. Execution success != correctness, and self-evaluation != independent verification.
His fix: strict separation of responsibilities — generation, execution in a controlled environment, external verification (builds, tests, static checks), separate failure classification, and evidence receipts recording what ran and what remains unverified. The model topology is not the trust boundary.
Across a 302-unit cross-provider paired evaluation, an adoption-gated setup showed positive deltas on HotpotQA, MATH 500/AIME, and MMLU, but negative same-model cases made him skeptical of unconditional debate or ensemble loops. Target architecture: single agent → external verification → escalate only when evidence is insufficient. He's packaging this into AdaptOrch (disclosed as his own product).
More from coding & agent
- Hindsight ships 0.8B model for agent memory decisions: 2.5x cheaper, 1.9x faster — victor_explore · 2026-11-04
- Human Oversight Is 70-75% of AI Agent Costs in Banking, McKinsey Finds — mikeflache · 2026-11-04
- Over 80% of Major Financial Institutions Now Run Agentic AI in Production — mikeflache · 2026-11-04
- Mnemos MCP nears launch after shared memory layer fix — RileyRalmuto · 2026-11-04
- Moving mobile dev to remote machines makes it multiplayer across product, design, engineering — rudrank · 2026-11-04
- Kuaishou's EvoRec: self-evolving agentic recommender lifts ad revenue 4.6% — _reachsumit · 2026-11-04