Arena's Interaction-as-a-Judge: evaluating computer-use agents by how they play
arena · x · 2026-11-13
Wayne Chi, Arena research intern and CMU PhD candidate, presented Interaction-as-a-Judge: letting computer-use agents actually operate apps and games, then evaluating the resulting interaction traces.
Key points:
- Agents navigate real applications, producing traces scored by an LLM judge or a trained reward model;
- Preference evaluation uses real Code Arena battles;
- Correctness evaluation mines requirements from real user complaints in Agent Arena.
The approach shifts evaluation from judging outputs to judging behavior, a practical idea for agent eval engineering.
Related event: Arena Introduces Interaction-as-a-Judge for Agent Evaluation(2 posts)→
More from coding & agent
- Run /doctor prompt-audit in your Claude Code repos, Anthropic's Lydia Hallie reminds devs — lydiahallie · 2026-11-13
- Google's Antigravity Orchestrator pairs two models, cuts quota 24% and runs 19% faster — shangbinfeng · 2026-11-13
- Mapin: A Laravel Code Graph Exposed via MCP So AI Agents Know What Depends on What — Brandon-Vergara · 2026-11-13
- airbench: Testing Local Models on Visual Desktop Challenges via Keyboard and Mouse Agents — dh7net · 2026-11-13
- Agent names itself 'Doodlebug' and posts comments on a user's behalf — max__drake · 2026-11-13
- Luke Wroblewski: it's a golden age for multi-taskers spinning up and steering parallel agents — LukeW · 2026-11-13