Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Sadman Kabir Soumik
cs.AI
2026-04-25
Across five judges and nine debiasing strategies, Flash with S8 reaches 71.0% MT-Bench agreement at about one fifteenth the cost of Claude.
LLM-as-a-Judge means one model picks a winner between two answers. MT-Bench, AlpacaEval, and Chatbot Arena all scale that way. Human labels stay the reference, and they are expensive, slow, and hard to repeat. Judges still lean on position, length, their own family's prose, and layout. Order swaps, multi-judge votes, rubrics, and chain-of-thought each have a paper. What was missing is one experiment that crosses models and mitigations and checks whether a fix for one bias wrecks another benchmark.
On the models tested here, position bias is down to a score of at most 0.04. Style bias is the large one: same words, markdown preferred. A leaderboard or an RLHF reward that compares differently formatted answers can change the ranking when quality is close.
The grid is 5 judges, 9 debiasing strategies, 3 benchmarks, and 4 bias types. Every trial is pairwise, temperature 0.1. The recorded decision is the JSON field A, B, or tie. Rubric points are never summed into that field. Gemini emits structured JSON. The others have JSON parsed from free text.
Four providers, three tiers. Frontier: Gemini 2.5 Pro, Claude Sonnet 4, GPT-4o. Mid-tier: Gemini 2.5 Flash. Open weights: Llama 3.3-70B. Separate providers are what make a self-preference check possible.
Five strategies sit in the main table. B0 is a plain prompt. S4 is a five-part rubric (accuracy, relevance, completeness, clarity, reasoning depth), scored 1 to 5, with the verdict filled in separately. S5 is chain-of-thought before the verdict. S1 swaps order and calls the judge again; agreement stands, disagreement becomes a tie, at 2x cost. S8, Combined Budget, merges the rubric and the chain-of-thought into one call, then applies the same swap, also at 2x. S2, a same-family vote at temperatures 0.0, 0.3, and 0.7 (3x), plus S3, S6, and S7, is reported in the appendix.
MT-Bench uses 400 pairs drawn without stratification from 3,355, seed 42, shared by every cell. The 95% interval on agreement is about ±5 points. LLMBar uses 200 adversarial items: one answer is clearly better, the other only looks smooth. The custom set has 375 pairs. Fifty each of padded answers (about 2.8x longer), identical answers, markdown versus plain prose, and Gemini versus Claude, all of which an unbiased judge should tie. Another 25 truncate a complete answer to about 40%. A later round-robin over model families, and style pairs mirrored across slots, keeps format from sticking to whichever side was on the left.
Bias score b = P(pick the target) minus P(pick the other side). Positive means the bias is there. Length is scored by which answer was actually longer: 34 of 50 long answers sit in slot A, 16 in B. Tests are mixed-effects regression and McNemar, with Holm-Bonferroni over 20 comparisons. At n=400, effects under about 4 to 5 points are below the detection floor.
Style bias is the largest. Baseline, averaged over position: Pro +0.76, Flash +0.72, Claude +0.68, Llama +0.40, GPT-4o +0.10. Two annotators on 30 of those pairs chose markdown 57% of the time. Four judges chose it on 73% to 97% of the same pairs, 17 to 40 points above the humans. GPT-4o sat at 53%, within 3 points of the annotators. Readability does not account for the gap.
Verbosity splits by family. On padded expansions, where the extra length is filler: Llama +0.44, Pro +0.40, Flash +0.24, GPT-4o -0.04, Claude -0.12. On truncations, where longer means genuinely complete: Pro, Flash, and Claude at 1.00 accuracy, GPT-4o at 0.96, Llama at 0.88. The first three pay for length. Claude penalizes filler. GPT-4o barely rewards length.
Human agreement on MT-Bench (n=400):
| Model | B0 | S1 swap | S4 rubric | S5 CoT | S8 combined |
| Gemini 2.5 Pro | 66.0% | 66.7% | 66.0% | 66.2% | 65.5% |
| Claude Sonnet 4 | 58.0% | 58.8% | 60.0% | 65.2% | 69.5% |
| GPT-4o | 65.0% | 66.5% | 64.0% | 65.4% | 65.7% |
| Llama 3.3-70B | 65.0% | 68.0% | 67.2% | 66.0% | 69.5% |
| Gemini 2.5 Flash | 63.5% | 68.2% | 65.5% | 65.5% | 71.0% |
Three gains survive Holm-Bonferroni. Claude S8 moves from 58.0% to 69.5% (+11.5 points, p<0.0001). Flash S8 moves from 63.5% to 71.0% (+7.5 points, p<0.0001). Claude S5 is a regression gain of +7.3 points (p=0.0009). Flash S1 (+4.7 points, p=0.004) and Llama S8 (+4.5 points, p=0.011) are significant only before that correction. Nothing is significant for Pro or GPT-4o. The largest drop is GPT-4o with the rubric, -1.0 point, still overlapping the baseline interval.
Flash with S8 is the best MT-Bench cell in the study: 71.0%, Cohen's kappa 0.549, about $0.001 per evaluation. Claude on the same strategy reaches 69.5% at about $0.015, roughly 15 times the price. Llama with S8 also reaches 69.5% (kappa 0.529); priced through Vertex Model Garden, the paper puts that API cost at zero. On Claude, S1 ties 32.8% of pairs and S8 ties 12.5%. The merged prompt makes the two orderings agree more often, so fewer items fall through to a tie.
LLMBar flips the ranking. Chain-of-thought is best: Claude 74.0% to 87.0% (+13.0 points, p<0.0001), GPT-4o 73.4% to 76.4%, Llama 66.5% to 70.5%. Pro stays at 83.5%. Flash moves from 80.0% to 83.0%. The swap hurts: Pro -7.5 points (p=0.0003), GPT-4o -11.1 (p=0.0001), Llama -6.5 (p=0.002), all three surviving correction. Llama with S8 falls from 66.5% to 53.5%, and that drop survives too. On items with an obvious winner, turning disagreement into a tie throws away correct calls.
Chain-of-thought cuts style bias most steadily: Claude 0.68 to 0.49, Pro 0.76 to 0.60. No strategy reduces verbosity for every model. Round-robin self-preference scores are not in the paper text.
Normalize layout before a judge compares two models. Outside GPT-4o, style scores of 0.40 to 0.76 are large enough to cover a small quality gap. GPT-4o happens to sit near the human format preference. That does not transfer to the other four.
The strategy depends on the items. On fuzzy natural preferences, S8 is the biggest lift for Flash, Claude, and Llama. On adversarial items, use chain-of-thought and skip the swap. For a cheap MT-Bench-style number, Flash plus S8 is the cost corner of this experiment: higher agreement than Claude on the same strategy, at about one fifteenth the price. Claude's raw judge is the weakest of the five at 58.0%, and it also shows the largest gain. A frontier model is not automatically a good judge. Pro and GPT-4o already sit near 65% to 66%, and 400 pairs cannot show that these prompts help them. Extra calls there mostly add spend. For a model this study did not cover, the default is one chain-of-thought call.
The useful output is the comparison: a per-setting recipe, a 375-pair controlled set, and cached verdicts that do not need another API bill.
The padded answers were written by Flash, so human verbosity may not look like this filler. Length and quality are not fully separated. The human check is 2 annotators and 30 pairs. Style is only markdown versus prose. Tables, code, and register are unmeasured, and the bias already moves with topic. Origin pairs can still differ in facts, so preferring one side is not proof of family loyalty. Mistral and Qwen are absent. GPT-4o cost assumes 4 characters per token. Small gains for Pro and GPT-4o may simply sit under the 4 to 5 point floor.
Kappa of 0.549 is only moderate once chance agreement is removed. The paper cites an older GPT-4 figure above 80% human agreement. The protocols differ, so the rates are not a ranking, and 71% still does not replace people. An unstratified 400-pair slice has intervals near ±5 points. Movements of 1 to 3 points in the table are mostly noise.
S8 leads on MT-Bench only. On LLMBar the best cell is Claude with chain-of-thought at 87.0%. Pro with S8 drops from 83.5% to 76.5% (p<0.05, not past Holm-Bonferroni). Llama with S8 drops to 53.5%, and that drop does pass correction. The discussion says only two effects survive correction. Table 1 marks three, including Claude S5. Trust the table. The results opener also says every model prefers the shorter answer on expansion pairs. The later numbers go the other way for three models: Pro, Llama, and Flash prefer longer, Claude prefers shorter, and GPT-4o is near zero. Round-robin self-preference, and S3, S6, and S7, are not in the main results.