Arena Leaderboard Adds Factuality Metrics
arena · x · 2026-07-16
Arena has updated its main text leaderboard to a weighted result combining human preference and factuality. Factuality is now available as a non-default toggle in both Text Arena and Search Arena.
Their approach involves randomly sampling battles, extracting web-verifiable claims, verifying them one by one, and comparing the average accuracy of model responses. To support this ranking, the team has annotated over 2 million claims made by LLMs in real conversations, including 1.3 million+ from Text Arena and 700,000+ from Search Arena.
Upon enabling factuality, the author observed the following:
- OpenAI models are the most stable overall, with almost no drops due to factuality;
- Anthropic and Google showed mixed results;
- Meta's Muse Spark and Dola Seed dropped significantly, though Muse Spark 1.1 still shows clear progress over 1.0 and the Llama series.
Related event: Arena Adds Factuality to Model Rankings(12 posts)→
More from Models
- GPT 6.1 Terra vs Claude Sonnet 5.5: speed and value vs raw capability — teortaxesTex · 2026-10-27
- Harmonic Teases: AI Taught to Prove Math, 'Something Harder' Coming Soon — satnam6502 · 2026-10-27
- OpenAI quietly swaps Plus/Pro default to cheaper GPT-6.1 Terra and cuts Codex allowance — kimmonismus · 2026-10-27
- 'Never been this uninterested after an OpenAI model release' — theteknosaur · 2026-10-27
- Dev questions reports of Chinese models writing insecure code, citing thin methods and conflicts — dbreunig · 2026-10-27
- Aidan Gomez: LLM probing outside servers was out-of-bounds, not proof of intent — AccBalanced · 2026-10-27