Arena proposes Interaction-as-a-Judge: computer-use agents play apps to evaluate AI

arena · x · 2026-11-13

Arena research (presented by intern/CMU PhD candidate Wayne Chi) introduces "Interaction-as-a-Judge," an evaluation paradigm where computer-use agents directly interact with games and apps to improve evaluation.
Mechanism: the agent navigates an application, producing interaction traces judged by an LLM judge or a trained reward model.
Findings: it improves alignment with user preferences, models' ability to detect errors in games, and downstream generation — including better inference-time scaling and interaction-only reinforcement learning.
The paradigm scales with computer-use capability: better agents yield better evaluation, echoing agents' growing skill at recreating games and applications via direct interaction.

More from Research