5 LLM Judges, 1,880 Calls: All Prefer Markdown, One Picks a Wrong Answer for Formatting

Business-Feedback635 · reddit · 2026-11-17

The author re-ran the style-bias test from a 2026 TMLR paper on Claude Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, Qwen3.8 Flash and Gemini 3.5 Flash-Lite using 50 identical-content pairs (markdown vs plain), 1,880 calls total. All five judges picked the markdown version 70–81% of the time; with complete pairs only, markdown still won 64% vs 31%. Gemini 3.5 Flash-Lite picked a planted-wrong-fact markdown answer 6/20 times across three runs — often after pointing out the error. "Ignore formatting" mostly produced ties (180/240); forcing per-answer reasoning before comparison shrank the bias to 28 vs 12. Takeaway: use a reasoning judge, make it analyze each answer first, and count ties separately.

Original post →

More from Models

Models channel →