Frontier Model Council Study: Models Fail Together
soumitrashukla9 · x · 2026-07-14
The authors re-ran frontier model experiments using the same initial benchmarks, including GPT 5.5, Gemini 3.1 Pro, Claude Opus 4.8, Grok 4.5. They developed an "error correlation" metric to measure the probability of two models answering incorrectly simultaneously.
They found:
- No council structure surpasses its strongest member on any benchmark.
- Models rarely disagree; across all 329 questions, there was never a case where "only one model got it right."
- When they answer incorrectly, they tend to fail together.
The author believes this supports the "consensus machines" argument: sharing training data, distillation targets, and RLHF habits causes models to converge on the same wrong conclusions. The text also notes that 6 pairs of vendors exhibited similar phenomena on MMLU-Pro Math.
More from Research
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Ben Recht: The world needs a truly open language model, and it's easier to build than you think — beenwrekt · 2026-10-26
- New Blog Breaks Black-Box Distillation Into Five Practical Steps — chrisrohlf · 2026-10-26
- DetectorScope turns black-box AI text detectors into auditable ones with sparse autoencoders — _akhaliq · 2026-10-26
- OPST paper shows LLMs self-improve with zero human data, guided only by topic descriptions — _akhaliq · 2026-10-26
- Tencent's HiSTEVE 1.0: hierarchical VLA agent plays Minecraft with native keyboard and mouse, hits 85% SOTA — _akhaliq · 2026-10-26