The Leaderboard Illusion
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, Sara Hooker
cs.AI, cs.CL, cs.LG, stat.ME
2025-04-29
An audit finds Chatbot Arena tilted three ways: providers test variants privately and keep only their best score (Meta ran 27 before Llama-4), proprietary models harvest far more battle data than open ones (Google 19.2% versus 8.9% for all open models), and 205 of 243 public models were silently deprecated.
Chatbot Arena has become the de facto standard for ranking LLMs: any user sends a prompt, votes between two anonymous model replies, and the platform folds pairwise outcomes into win-rate scores with a Bradley-Terry model. It shapes media coverage, funding, and procurement, so the stakes are high. The open question is whether the arena is actually level. The author team (led by Cohere Labs, with Princeton, Stanford, Waterloo, MIT, AI2, and the University of Washington) has an unusual vantage point: some of them submitted Cohere models to the Arena, and through long-run crawling plus partial access to private testing they could see anonymized variants the public never sees. They ask whether the rules systematically favor a few players.
This is an audit paper, so "method" is how the distortions were surfaced. Several evidence streams are combined: a crawled snapshot of the Arena from January to March 2025 that covers both public models and anonymized private variants; provider attribution by prompting anonymous models to self-identify (an approximate method, with codenames published for peer correction); estimation of battle share and sampling rates from public stats; a controlled training experiment that fine-tunes on 0% to 70% Arena data and measures ArenaHard win-rate; and a comparison of model deprecations against the official deprecated list in the FastChat backend. Goodhart's law frames the argument: once a metric becomes a target, it gets gamed.
The first distortion is private testing plus selective disclosure. A few providers (Meta, Google, OpenAI, Amazon) can test many variants privately before launch, retract scores, and submit only the best. Meta tested 27 private variants in the month before the Llama-4 release. Simulations show that picking the best score out of N variants systematically inflates the Arena rating. The version on the public leaderboard also need not match the public API.
The second distortion is sharply unequal data access. By share of total battles, Google sits near 19.2% and OpenAI near 20.4%, while 41 fully open-source models combined reach only about 8.9%; 61.4% of all data flows to proprietary providers. Proprietary models are sampled more often, while open-weight and open-source models are under-sampled and removed more frequently.
The third distortion is that Arena data plainly lifts scores. Raising the share of Arena training data from 0% to 70% more than doubles the ArenaHard win-rate, from 23.5% to 49.9%, with relative gains up to 112%. This is a conservative estimate, since some providers have access to five to ten times more private API data.
The fourth distortion is that deprecation breaks rankings. Of 243 public models, 205 were silently deprecated, while the FastChat backend officially lists only 47. Among the silently deprecated, 64% are open-weight or fully open-source. Silent deprecation erodes the connectivity of the comparison graph, undermining the Bradley-Terry assumption and making the scores of remaining models unreliable.
The authors offer actionable reforms: prohibit score retraction after submission; cap the number of private variants per provider; set auditable deprecation criteria (stratified across proprietary, open-weight, and open-source, retiring the bottom 30th percentile within each category once rankings converge); actually deploy the active-sampling rule from their own paper; and publish quarterly detail on tested, deprecated, and sampled models.
For practitioners: do not treat Arena rank as ground truth for model quality, especially when comparing a freshly launched proprietary model against an older open one. Rank mixes in overfitting to the Arena distribution and data-access privilege, not just capability. For the field, this is a measurement-integrity warning with cheap, actionable fixes. A fair caveat: the Arena is a valuable platform built by organizers and an open community, and the ask is reform, not abandonment.
The authors' own caveats are substantial. They lack access to Arena's raw data (the platform deduplicates, removes suspicious votes, and strips private battles), so adversarial voting cannot be studied. The crawled snapshot covers only January to March and coincides with the Llama-4 launch, so Meta's count is likely inflated and other providers underestimated. The training experiment uses a fraction of the data real providers hold, so true overfitting is probably understated. Provider attribution relies on model self-identification and carries error. One more point the authors understate: the 112% and doubling gains come from training on Arena data and testing on ArenaHard, an in-distribution test set, so part of what they measure is fitting the Arena distribution, which is exactly the authors' point, but the magnitude should not be read as general-capability gain. Some findings are also tied to the January-to-March window.