Catch LLM judge position bias with a zero-label A/B swap test

Beneficial_Use2116 · reddit · 2026-10-11

The author shares a ground-truth-free way to detect position bias in LLM judges: swap "Response A" and "Response B" and re-run — a correct judge picks the same answer either way; a flipped verdict is a self-contradiction.

Tested on 21 pairwise cases plus controls across two models:

Factual Qs and clear-winner pairs: 0 flips (both Haiku and Sonnet);
Genuinely close calls: Haiku 6/21, Sonnet 3–5/21, varying run to run;
Controls at 0 flips confirm the close-call flips are real position bias, not randomness;
The rate is noisy at n=21, so it's a detector and warning, not a precise ranking.

Fix: average both orders (or rotate positions for >2 candidates); a bigger model doesn't automatically fix it. The bookkeeping is packaged as pip install wobbly (judgebias()), 40 lines to script yourself.

Original post →

Catch LLM judge position bias with a zero-label A/B swap test

More from coding & agent