Your LLM Eval Is Lying: Formatting Swings LLaMA-2 Accuracy by 76 Points
irombie · x · 2026-10-30
A full write-up of the author's talk "Your LLM Evaluation Is Lying to You," with every claim sourced and demos runnable. Core gap: almost everyone can ship an LLM feature, far fewer can prove it works.
Key evidence:
- Models are distributions, not functions: Thinking Machines Lab (Horace He) sent the same prompt to Qwen3-235B 1,000 times at temperature 0 and got 80 different completions — all identical for the first 102 tokens, diverging at token 103 (992 said "Queens, New York," 8 said "New York City"). Batch-invariant kernels made all 1,000 identical, so the variation came from the inference server, not sampling.
- Formatting alone moves accuracy: Sclar et al. changed only meaning-preserving formatting (separators, spacing, casing) and LLaMA-2-13B accuracy moved by up to 76 points; typical spread across 50+ tasks was 10 points — still larger than most claimed "improvements."
- Same eval, same model, 5 runs: 73% to 93%. Rerun and report the spread; changes smaller than the spread are variance, not results.
Code is available (runs with --mock, no API key needed), and the author invites readers to test their own judges with judgeflip.py.
More from coding & agent
- Data leakage hit 648 papers across 30 fields; autoarxiv now auto-reproduces arXiv papers — pandeyparul · 2026-10-30
- METR trial: AI made experienced devs 19% slower while they felt 20% faster — srimisra · 2026-10-30
- Clairvoyance: free desktop tool wires Claude Code, Codex and other agents into your daily workflow — draginol · 2026-10-30
- Same Prompt, Same Data: Claude's Fabric App Beats Its Dashboard at One-Fifth the Cost — adnan_hashmi · 2026-10-30
- Antigravity's Google Slides Plugin Audits an 80-Slide Deck Against Docs — rseroter · 2026-10-30
- OpenAI hosts Cloud Native Hackathon Nov 9; hack turns M5Stack into a k8s desk pager — OpenAIDevs · 2026-10-30