Study: Cheap LLM Judges Make Correlated Errors, Cascades Barely Help

Delip Rao's team released an arXiv paper on Jev, a fast, cheap Flash-class decision model with 1-2 orders of magnitude speed/cost advantages. However, its errors are highly correlated with peer fast LLMs, making cascade-based cost-saving routing largely ineffective. This echoes prior ICML 2025 findings and is a substantive warning for architectures relying on cheap judges plus cascade escalation.

Confirmed

• Jev makes highly correlated errors with other Flash-class LLMs (e.g., DeepSeek).
• Key data: on Jev's most confident errors, 96.0% of LLM judge verdicts repeated the same wrong answer; independence would predict 50%.
• Cascade experiments: even with tuned confidence thresholds, cascading gains at most 1.5 points over the best single judge with cross-fitted thresholds, and 2.0 points with oracle thresholds; cascading to a jury-style judge ensemble also helps little.
• @srchvrs independently relayed the same conclusion: Jev is fast and cheap but its errors correlate with peer fast models, so routing low-confidence outputs to stronger LLMs yields limited benefit.

Why it matters

• Delip Rao argues Jev from typesafeai represents a lasting design paradigm for judge models, but nearly all Jev-like judges are fine-tunes of existing LLMs, so judges make similar errors and reinforce the same biases in production, even when switching or ensembling models.
• This directly echoes ICML 2025 work: Kim et al. analyzed 350+ models and found larger, more accurate models make highly correlated errors, even across vendors.
• Since errors share the same distribution, LLM judges largely reproduce Jev's most confident mistakes, and cascade routing offers little genuine judge diversity or quality lift.

2026-09-26 ~ 2026-09-27 · 8 related posts

• [source] Jev makes the same errors as flash-tier LLMs, undercutting cascade savings, new study finds — deliprao · 2026-09-26
• 96% of LLM verdicts repeat Jev's most confident errors, experiments show — deliprao · 2026-09-26
• [source] Experiments show LLM judge cascades gain at most 2 points as errors correlate — deliprao · 2026-09-26
• ICML 2025 study of 350+ models found highly correlated errors across providers — deliprao · 2026-09-26
• Delip Rao: finetuned judge models like Jev won't deliver true judge diversity — deliprao · 2026-09-26
• LLM judges are all finetunes of the same models, risking shared biases — deliprao · 2026-09-26
• [source] New arXiv draft: flash-tier LLM Jev makes correlated mistakes, cascading yields little — srchvrs · 2026-09-26
• Jev makes the same mistakes as flash-tier LLMs, so cascading them buys almost nothing — joecole · 2026-09-27