5 LLM Judges, 1,880 Calls: All Prefer Markdown, One Picks a Wrong Answer for Formatting
Business-Feedback635 · reddit · 2026-11-17
The author re-ran the style-bias test from a 2026 TMLR paper on Claude Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, Qwen3.8 Flash and Gemini 3.5 Flash-Lite using 50 identical-content pairs (markdown vs plain), 1,880 calls total. All five judges picked the markdown version 70–81% of the time; with complete pairs only, markdown still won 64% vs 31%. Gemini 3.5 Flash-Lite picked a planted-wrong-fact markdown answer 6/20 times across three runs — often after pointing out the error. "Ignore formatting" mostly produced ties (180/240); forcing per-answer reasoning before comparison shrank the bias to 28 vs 12. Takeaway: use a reasoning judge, make it analyze each answer first, and count ties separately.
More from Models
- Gemini Pro Users Report Model Selection Resetting to Flash Lite Despite Paying — GoddogMC · 2026-11-17
- "Microsoft polled the 8 people who use Copilot": Claude-vs-Copilot report becomes a meme — djcows · 2026-11-17
- NVIDIA vet on Nemotron: open models face real data restrictions, 'genuinely hard' to build — PavloMolchanov · 2026-11-17
- "No Taste but Genuinely the Best": Early Verdict on GPT 6 Astra — justalexoki · 2026-11-17
- Google expands Antigravity program to 10,000 scientists with free Gemini 4 Carbon access — GoogleDeepMind · 2026-11-17
- Google DeepMind unveils Gemini 4 Carbon for physics, math and life science research — GoogleDeepMind · 2026-11-17