2,611 real interviews show rater bias predicts AI agent scores better than actual performance
Exp_Mark · x · 2026-10-15
The Economics Research Unit at micro1 released its first paper, "Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program," based on 2,611 real production interviews with a live AI interviewing agent.
- Reviewer identity was a stronger predictor of scores than interview timing; a single overly strict rater could skew metrics — apparent agent progress may just reflect rater composition changes.
- The paper covers rater effects, drift, and the return of running a consistent evaluation program.
Core claim: most companies can't tell whether their AI agents are genuinely improving because evaluation metrics capture reviewer bias, not AI performance.
Related event: Study: Rater Bias and Drift Corrupt AI Agent Benchmarks Within Weeks(3 posts)→
More from coding & agent
- Every publishes hands-on guide to driving AI agents with your voice — every · 2026-10-15
- Maple ships Alchemy provider: declare Cloudflare Worker telemetry, alerts as code — samgoodwin89 · 2026-10-15
- Open-Agent-Email: free open-source alternative to AgentMail for AI agents — matchaman11 · 2026-10-15
- AWS Gen AI Weekly Redesign: One-Page Exec Summary, AgentCore Memory Goes Usage-Based — DigitalColmer · 2026-10-15
- Nous Research Launches 50+ Connectors for Hermes Agent, No Account Required — NousResearch · 2026-10-15
- Dev group chat: skip runtime LLM web search, index products via API instead — vxnuaj · 2026-10-15