micro1's ERU paper finds rater bias and 5-week drift make AI agent evals unreliable
Exp_Mark · x · 2026-10-15
The Economics Research Unit at micro1ai has released its first paper, "Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program," analyzing 2,611 real production interviews from a live AI interviewing agent.
- Rater effects mask real progress: reviewer identity was a stronger predictor of scores than interview timing. A single overly strict evaluator made a genuinely improving agent appear stagnant; once corrected, reliability metrics rose by half a standard deviation.
- Evals go stale: evaluation systems drift and lose validity after roughly five weeks.
The takeaway: most companies can't tell whether their AI agents are improving because their metrics reflect reviewer bias and drift, not model capability — a continuously running evaluation program is essential.
Related event: Study: rater bias makes AI agent benchmarks unreliable within weeks(2 posts)→
More from coding & agent
- Sentry CEO: AI agents still write messy code but are getting better at managing it — zeeg · 2026-10-15
- Omarchy bug: missing rtkit causes Chrome/Electron apps to get SIGKILLed — fix inside — mark_k · 2026-10-15
- 2,611 real interviews show rater bias predicts AI agent scores better than actual performance — Exp_Mark · 2026-10-15
- Google's DuoDiscover tops NVIDIA SOL-ExecBench, 2.68x faster than production TPU kernels — lateinteraction · 2026-10-15
- Agent stack check: harnesses, OTel telemetry and MCP are settled — evals remain a mess — _ScottCondron · 2026-10-15
- Claude Code's safety guardrails refuse even non-sensitive env var copies, user complains — Promptmethus · 2026-10-15