1MB replay script beats frontier models: NeurIPS oral paper exposes flaws in CUA benchmarks
proceduralia · x · 2026-09-29
A paper on evaluating computer use agents (CUAs) was selected for a NeurIPS oral presentation — only 112 orals out of 30,709 submissions.
Key finding: a 1MB replay script that blindly executes a recorded action sequence without ever observing the screen outperforms frontier models on prominent static benchmarks; the authors prove its expected success rate exactly equals the source agent's pass@k in deterministic environments.
Root causes: non-principled environment design (static, unsandboxed, unreliably verified) and flawed evaluation methodology (naive aggregation, misuse of pass@k for stateful UI interactions).
Contributions:
- PRISM, five design principles for CUA environments (privileged verification, realistic environments, integrity-checked configs, sandboxed execution, multifactorial variability), instantiated in DigiWorld — a benchmark of 15 realistic sandboxed mobile apps spanning over 3.2 million verified unique configurations
- An aggregation framework pairing Wilson score intervals with hierarchical bootstrap, producing confidence intervals that correctly account for the nested structure of CUA benchmarks
Related event: NeurIPS oral paper exposes replay-cheating flaw in CUA benchmarks(2 posts)→
More from Research
- Continual-ARC: lifelong ARC-AGI stream finds agent harness persistence rarely beats simple caching — unifyai · 2026-11-16
- DySENSE: MIT's GenAI framework synthesizes satellite imagery with dynamic city energy maps — MIT · 2026-11-16
- WebArena-Pro: a 300-task multimodal benchmark across 20 self-hosted apps for web agents — McGill-NLP · 2026-11-16
- Training-free page-sparse decode in llama.cpp: 1.72× faster 128k decoding on an RTX 5090 — topppggg · 2026-11-16
- Training Qwen3-8B to drive a simulated hand spawns a dedicated internal 'pressure' direction — repligate · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16