Study: Rater Bias and Drift Corrupt AI Agent Benchmarks Within Weeks

A first paper from micro1's Economic Research Unit, based on 2,611 real interviews, finds AI agent evaluations are corrupted by rater bias and drift: scores vary wildly with judging standards and lose validity within about five weeks, with rater effects outweighing agents' real progress.

2026-10-14 ~ 2026-10-15 · 3 related posts

• Same agent, same task, judged twice: strict scoring yields under 20% of human baseline — YvesMulkers · 2026-10-14
• micro1's ERU paper finds rater bias and 5-week drift make AI agent evals unreliable — Exp_Mark · 2026-10-15

1 near-duplicate retellings: Exp_Mark