LMArena open-sources 6,390-pair web app preference dataset, shows interaction-based judging beats code review
arena · x · 2026-11-13
LMArena released two open datasets and a paper on Interaction-as-a-Judge, where computer-use agents evaluate apps by directly interacting with them: ArenaWebdevPreferences (6,390 app pairs with human preference labels) and ArenaWebdevCorrectness (189 game versions, 362 human-verified requirements).
Key findings:
- Predicting human preferences improves 65.6% → 69.1% vs non-interactive judging; requirement verification 77.0% → 81.6%
- At inference time, 75.5% of broken requirements are repaired within five attempts
- Fine-tuning Qwen-3.5-9B with interaction-only rewards (no code access) gains +6.7pp win rate (+50 Arena points) over the base model; vs code-only rewards it raises app success from 81.3% to 88.5% and cuts code bloat 35.7%
Team includes Wayne Chi, Ion Stoica, Anastasios Angelopoulos, Ameet Talwalkar and Cho-Jui Hsieh.
More from coding & agent
- Stanford Launches CS329Z: Engineering AI Agents, Taught by DSPy and SWE-bench Authors — stanfordnlp · 2026-11-13
- JoMoHa jointly RL-trains model and harness, hits 97.0% on ALFWorld — omarsar0 · 2026-11-13
- YC F26 team open-sources runtime to run 10,000 agents in one session — KlausCodes · 2026-11-13
- Open-Source Pipeline Turns LiDAR + OpenStreetMap into Isometric Pixel-Art Cities with FLUX.2 — zsakib_ · 2026-11-13
- X, MIT, UW & Stanford Launch WSDM Cup for AI Agents Writing Community Notes, $10k Prizes — NathanpmYoung · 2026-11-13
- Benchmarks won't move users off Claude Code — Antigravity needs real product uplift, dev argues — Justin_Halford_ · 2026-11-13