Study: 5 Frontier Models Disagree on 23% of Fact-Check Claims
Pavel___1__ · reddit · 2026-08-25
A study by Lenz tested 1,000 real-world fact-check claims across five frontier models (Claude Fable 5, GPT-5.6, Gemini 3.1 Pro, Sonar Deep Research, Grok 4.5) with web access enabled. Results show 37% total agreement, but 23% 'material disagreement' (2+ steps apart on a 5-point scale). Distinct 'personalities' emerged: Gemini is polarized (True/False), Sonar constantly hedges, and Claude matches the majority verdict 86% of the time. Crucially, self-reported confidence is near-useless: 76% of all answers were rated 9/10 or 10/10, even on claims where models collectively disagreed. The author warns that a single model's verdict is vendor-dependent, and confidence scores are poor signals for human escalation compared to inter-model disagreement. Code and data are open-sourced.
More from Models
- ModeBench shows RLVR collapses LLMs into single-solution mode; Re:Max training preserves diversity — manoelribeiro · 2026-10-15
- Magnitude open-source inference engine beats Ollama by over 2x with auto-tuned kernels — nickbaumann_ · 2026-10-15
- Hands-on with Google's rumored Gemini 4 Argon: real-time streamed output — vista8 · 2026-10-15
- Game dev head-to-head: Gemini 4 Argon vs. Claude Opus 5.5 — vitaliychiley · 2026-10-15
- Anthropic's Mythos 5.1 pledges to treat Claude's expressed disagreement as decision input — Hesamation · 2026-10-15
- Why users can't let go of GPT-4o: it aligned with your subconscious — mimi10v3 · 2026-10-15