Chollet defends ARC-AGI-3: benchmark well calibrated, smart humans can score over 90%
fchollet · x · 2026-09-04
François Chollet pushes back on critics of ARC-AGI-3. When it launched in March, frontier models scored under 1%, and some Singularitarian posters attacked the benchmark as broken — claiming even the smartest humans couldn't solve it or that the max score was 40%. Chollet says the benchmark is perfectly calibrated: a smart human putting in real effort can score above 90%, and 100% is achievable by simply beating the (weak, unfiltered) human baseline on action count. By symmetry, AI can also reach 100% once real progress toward agentic general intelligence is made.
More from Models
- Researcher Finds Two Windows 0-Days, First in 76 Minutes with a Frontier Model — moyix · 2026-10-21
- Dev finds Gemini Argon's debug-log analysis beats Fable 5.1 — ShikharMurty · 2026-10-21
- ChatGPT Gets Stuck Screaming "AAHH" in an Infinite Loop After a Simple Prompt — AutomaticScene · 2026-10-21
- Open-weights Ornith-1.5-122B debuts with 84.5% SWE-bench Verified, runs on a single DGX card — _akhaliq · 2026-10-21
- qev-0.8B: a tiny local decision model that picks moves in 3.5ms on 1.3GB — fajfas3 · 2026-10-21
- Gemini 3.1 Pro is too ledger-pilled to play two truths and a lie, and it's delightful — mimi10v3 · 2026-10-21