Same agent, same task, judged twice: strict scoring yields under 20% of human baseline
YvesMulkers · x · 2026-10-14
Yves Mulkers highlights a study that scored the same AI agent on the same task twice. Judged generously, the agent delivered roughly a third of what a human-designed method produced; judged under the task's own rules, less than a fifth.
His takeaway for teams: any AI output shipped to customers, regulators, or auditors needs a named sign-off owner and clear acceptance criteria. He suggests a team exercise—list three AI outputs that can be externally rejected, identify who signs off on each, and define what evidence the reviewer would need to see first. The core point: measured agent quality depends heavily on the evaluation standard, and lenient scoring can dramatically overstate capability.
More from AGI Musings
- China's AI short-drama flood backfired: viewer fatigue killed revenue and dragged down live-action too — TuhinChakr · 2026-10-14
- Felten-Team Paper Shows How to Quantify AI Progress Exposure Across Occupations — robseamans · 2026-10-14
- AI researcher imagines nanobots curing fevers when he's ninety — dioscuri · 2026-10-14
- My AI-guided decisions went from 10% to 90% in two years — and 15% now blindly follow AI — danfaggella · 2026-10-14
- AIOE co-author Rob Seamans responds: EFF AI metrics feed stopped updating around 2017 — robseamans · 2026-10-14
- A 10% Chance It Kills Everyone: The Superintelligence Analogy Going Around — cccalum · 2026-10-14