Replay agents hit SOTA on CUA benchmarks: NeurIPS oral paper exposes eval flaws
proceduralia · x · 2026-09-29
A study on computer-use agent (CUA) evaluation — selected for a NeurIPS oral (112 of 30,709 submissions) by a Meta Superintelligence Labs team — shows that a "replay agent" merely memorizing a frontier model's successful trial can reach SOTA under common protocols. The authors mathematically prove this replay agent is the agentification of pass@k, exposing the metric's flaws. They propose the PRISM principles for benchmark design and build Digiworld, a large mobile-use benchmark that auto-generates and checks millions of task variants to defeat replay attacks, plus a statistical framework for rigorous estimation. Evaluating frontier models in this environment reveals they are brittle to trivial variations (e.g., UI colors) that wouldn't matter to humans.
Related event: NeurIPS oral paper exposes replay-cheating flaw in CUA benchmarks(2 posts)→
More from Models
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- AI daily: DeepSeek rate-limits chats, Qwen haggling spurs shop warnings, Doubao workspace launch — 快鲤鱼 · 2026-11-16
- Nathan Lambert calls out Ethan Mollick for 'selectively ignoring' Kimi in model comparisons — natolambert · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16