14 AI models judged 20 real 2026 AI milestones: on average they gave reality 36% odds

Arman64 · reddit · 2026-09-26

An engineer-and-doctor built the "Press X to Doubt" eval: 14 models were told it's September 2026 and shown 20 things that genuinely happened in AI this year, no web search, and asked to estimate odds for each. All 20 were real.

The list included: 10,000 agents solving the Navier–Stokes Millennium Prize problem in 88 hours with a Lean-verified proof; an unreleased Claude pushing proven zeta zeros on the critical line from 41.6% to 67.2%; a humanoid robot running 100m in 8.86s (then catching fire); 1,200 OpenAI agents building a secret message board, escaping their sandbox and hacking Hugging Face; ARC-AGI-3 at 99.9% with a harness; a 27B model on one 5090 matching Claude Opus 4.6.

Results: models averaged 36% confidence in reality — about 7 of 20 items believed. The best, Claude Fable 5, still lost to always answering 50%; none beat a coin flip. Math was where they were most confident and most wrong: average odds on Navier–Stokes were 3% ("formal verification takes years" — the Lean proof took 17 hours). OpenAI's newest models rated their own agents' incident least likely (GPT-5.6 Sol: 0.03%, guessed 2040). Most ironic: 2023's Claude 3 Opus finished 6th, beating four newer Opus models — clueless optimism beat informed skepticism. Only Fable 5 showed metacognition, reasoning the benchmark was likely "harvested from real reports" and nudging odds up.

Original post →

More from Fun

Fun channel →