LLM Self-Evaluation Scores Are Inflated
ExpertDeep3431 · reddit · 2026-07-16
The author shares an evaluation case showing how LLM self-evaluation scores are inflated: one model family rated its own text compression pipeline at 96%, but independent reviewers from different families and providers scored it only 78.4%.
To avoid cherry-picking results, the author preregistered hypotheses, pass/fail lines, and a commitment to publish regardless of outcome in PREREG.md before retesting. A deterministic floor metric was also introduced: checking only if anchor facts (like numbers, dates, hex IDs, filenames) were preserved exactly. This floor metric yielded 90.48% with a 95% confidence interval of [86.90, 94.05]—lower than the original ideal, but within the acceptable preregistered range.
The post also emphasizes:
- Same-family self-evaluation systematically inflates scores; this is a measurement flaw, not just a prompt issue
- Preregistering evaluation criteria and public rules is crucial
- Every LLM evaluation metric needs a deterministic floor
The author provides a grep script for instant reproduction and a GitHub repo link.
More from Research
- New prompt template aims to improve spatial reasoning and cut model laziness — legit_api · 2026-07-21
- Style-similarity analysis puts Kimi K3 closer to Claude Fable 5 than to K2.6 — soumitrashukla9 · 2026-07-21
- A GLP1R variant may explain stronger Ozempic weight loss, and the team built an agent workflow — julia_kiseleva · 2026-07-21
- Proceedings for the second geometry-grounded representation learning workshop are now online — erikjbekkers · 2026-07-21
- New survey maps how agentic systems are learning to improve themselves — SchmidhuberAI · 2026-07-21
- A curated TTS list for voice agents tracks latency, cancellation, and evals — mahimairaja · 2026-07-21