GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently
nitarshan · x · 2026-10-16
Jasper Dekoninck and colleagues used GPT-6.1 Sol to systematically check whether main theorems in arXiv math papers are actually correct.
Findings:
- 6% of papers before 08/2026 contain main theorems that are false
- Only 1.5% of papers from the last two months contain similar mistakes
The chart breaks down severities: for pre-Aug 2026 papers (n=656), 35.7% had no counterexample found, 27.9% had literal gaps, 30.5% had errors with simple repairs, and 5.9% needed central claims reworked; all categories improved sharply for Aug–Sep 2026 papers (n=137).
Either recent papers are genuinely better, or frontier models are increasingly unable to fault them — both readings are notable.
More from Models
- Microsoft unveils MAI-Base-2: 2.8T-param MoE, 4-12x more pre-training FLOP-efficient than DeepSeek — ChengleiSi · 2026-10-16
- Pro Users Say They Can't Access Gemini 4 Argon, Call Antigravity 'Nearly Unusable' — realmeetjames · 2026-10-16
- Tiny classifier face-off: Vega runs on CPU, phone, browser at 75ms on M5 — alexcovo_eth · 2026-10-16
- Paper AI Tigers: Chinese frontier LLMs are 3-6x cheaper yet only 19% of OpenRouter use — gleech · 2026-10-16
- Cisco's Antares AI model ranks risky files for security teams, scores 0.823 on source but 0.000 on binaries — shashib · 2026-10-16
- Stefano Ermon teases 'largest and most capable model' landing next week, opens early access — StefanoErmon · 2026-10-16