Arena CEO: AI Evaluators Can't Keep Up, So Policy Should Leave Room for Competition
MattPerault · x · 2026-11-17
In the a16z AI Policy Brief podcast, host Matt Perault speaks with Arena cofounder and CEO Anastasios Angelopoulos about evaluating AI through real-world use.
- Arena began as a UC Berkeley research project comparing chatbot responses and grew into a platform measuring how models perform on real work tasks.
- They discuss Arena's Alignment Index, which measures how often models misbehave—acting without permission or reporting unfinished work as done.
- The conversation covers what such measurements can and cannot tell us about capabilities and risk, and how competition on an Index score can push developers to improve.
- Central argument: evaluation methods must keep evolving, and policymakers should leave room for new evaluators and better methods rather than locking in a handful of approved approaches or incumbents.
- Implications for policymakers (better capability/risk judgments), businesses (model selection), and builders (room for new measurement approaches).
Related event: LMArena Founder: Policy Must Leave Room for New AI Evaluations(2 posts)→
More from AGI Musings
- AI raised the ambition floor: researcher now lets optimization runs continue past the first win — generativist · 2026-11-17
- Why the AI safety movement is hard to beat: it lures Jane Street traders to $60k Hill jobs — panickssery · 2026-11-17
- CAIS and Scale AI Launch Remote Labor Index to Measure AI Automation of Remote Work — ben_j_todd · 2026-11-17
- Ben Todd extrapolates remote-work automation: odds doubling every 2 months, near-100% by 2028 — ben_j_todd · 2026-11-17
- Fields Medalist: AI Has Hit Math at Full Throttle, and Mathematicians Are Struggling — haider1 · 2026-11-17
- Oxford, DeepMind, OpenAI & Anthropic paper argues safe AI isn't the same as good AI — alex_verem · 2026-11-17