Arena Introduces AutoEval: Minute-Level Ratings via Reward Models
arena · x · 2026-07-31
Arena's official blog announced the introduction of AutoEval to its leaderboards to address the time-consuming nature of collecting real human votes.
- How it works: It trains a Reward Model (RM) on Arena's massive human preference dataset to automatically cast proxy votes. Combined with live human votes, this unlocks a model ranking in under an hour.
- Multimodal support: The mechanism extends beyond text to vision, image generation, and code. For instance, its text-to-image RM was trained on over 3 million preference pairs, achieving SOTA performance on the public MMRB2 benchmark.
- Dynamic updates: AutoEval scores provide a Day-1 signal for new model launches and are clearly labeled, later being updated and validated as sufficient human votes accumulate.
Related event: LMSYS Introduces AutoEval: Minute-Level Evaluation via Reward Models(5 posts)→
More from Models
- Same debate, with chart: can harnesses sustain AI progress past the plateau? — Otherwise-Report1848 · 2026-10-16
- Mistral touts STEM benchmark win, but leaderboard shows it ranking sixth overall — MistralAI · 2026-10-16
- Local tool calling compared: how Gemma 4, Llama 3 and Mistral differ — leslysandra · 2026-10-16
- $2 Gemini Pro accounts flood the gray market, spawning waves of AI spam comments — huangyun_122 · 2026-10-16
- Haider: Opus 5.5 vs Astra on SWE tasks — Astra has a higher floor, reasoning in a tier of its own — haider1 · 2026-10-16
- Artificial Analysis launches Decision Index for the emerging decision model category — ArtificialAnlys · 2026-10-16