Every LLM ranks itself #1 on self-generated benchmarks, EMNLP 2026 paper finds

shangbinfeng · x · 2026-09-05

A study accepted to EMNLP 2026 Main (arXiv:2509.26600) deconstructs self-bias in automated LLM benchmarking, where a model generates the testset and grades the outputs.

Using machine translation as the primary testbed, bias arises from two compounding sources — LLM as a testset and LLM as an evaluator — and their combination amplifies the effect.
Even with explicit diversity controls, each model's implicit stylistic tendencies produce homogeneous, model-specific data that inflates its own scores.
The bias is strong enough that every model ranks itself #1, overriding peer-consensus ordering; the phenomenon extends to open-ended generation on Chatbot Arena-style tasks.
The authors' proposed diversity metric partially mitigates but does not eliminate the bias.

Original post →

Every LLM ranks itself #1 on self-generated benchmarks, EMNLP 2026 paper finds

More from Models