LMSYS Introduces AutoEval: Minute-Level Evaluation via Reward Models
LMSYS (Arena team) officially announced the introduction of the AutoEval mechanism in the leaderboard to address the time-consuming process of collecting real human votes. The system trains specialized reward models (RMs) on massive real user preference data, providing Day-1 rapid evaluation signals for newly released models, with updates after sufficient human votes are accumulated.
Confirmed
- How it works: AutoEval trains reward models on millions of real Arena user preferences, generating proxy votes automatically for second-level evaluation. Specialized reward models are trained for different modalities (e.g., vision, image generation, code).
- Performance: In text-to-image tasks, using preference data beyond a certain scale, it outperforms the baseline by 9 points.
- Validation: The team compared AutoEval's early rankings with the final live leaderboard. When score gaps between two models are large, AutoEval's early ranking accuracy exceeds 90%.
Why it matters
AutoEval drastically shortens the model evaluation cycle, allowing new models to receive high-quality, real-data-calibrated evaluation signals on their first day. This improves leaderboard update efficiency and provides developers with reliable candidate checkpoint comparisons before the full leaderboard is officially released.
2026-07-31 ~ 2026-07-31 · 5 related posts
Primary sources
- [source] Arena Validates AutoEval: Over 90% Accuracy in Ranking Models — arena · 2026-07-31
- [source] Arena Introduces AutoEval: Minute-Level Ratings via Reward Models — arena · 2026-07-31
- [source] Arena Details Multimodal Reward Models, Beating Baselines by 9 Points — arena · 2026-07-31
- LMSYS Launches AutoEval: Ranking Models via Reward Models in Hours — vista8 · 2026-07-31
1 near-duplicate retellings: arena