LMSYS Launches AutoEval: Ranking Models via Reward Models in Hours
vista8 · x · 2026-07-31
LMSYS has officially launched AutoEval, a new evaluation methodology that ranks models using reward models trained on millions of real Arena user preferences.
Highlights:
- Provides high-quality evaluation signals calibrated on real preference data.
- Shows strong alignment with live human evaluations.
- Delivers evaluations that are several orders of magnitude faster (taking hours instead of days).
- Supports Text, Vision, Image, and Code Arena.
AutoEval enables the platform to evaluate newly launched models much faster and share results with the community sooner. Estimated scores for new models will now be displayed directly on the leaderboard.
Related event: LMSYS Introduces AutoEval: Minute-Level Evaluation via Reward Models(5 posts)→
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16