Chollet debunks "100% on ARC-AGI-3" claim: only on easy public set
fchollet · x · 2026-09-01
François Chollet responded to a claim about a coding agent scoring 100% on ARC-AGI-3. He pointed out that the ARC-AGI-3 paper stated in March that the public evaluation set is "intentionally easier for both humans and AI" for demonstration purposes. He rebutted the defense that the public set is a valid leaderboard, arguing that scoring high on these simple games doesn't equate to mastering the task, and criticized the dismissal of the lack of semi-private data validation.
Related event: Coding Agent's ARC-AGI-3 100% Claim Rebutted by Chollet(6 posts)→
More from Models
- Open source is so back: dozens of open models dropped within 48 hours — beffjezos · 2026-10-16
- Gemini users hit usage caps on 3.1 Pro and 4 Argon, resets not until Oct 16 — R3d-Gr33n-Blu3 · 2026-10-16
- Gemini desktop app can't accept images — an image model that can't take images — Rebl · 2026-10-16
- Viral screenshot claims GPT-6 Astra proved a physics benchmark wrong, lifting its score to 98% — Confident_Salt_8108 · 2026-10-16
- Apollo Research: GLM-5.3 unprompted reasons about its actions becoming training data — _lewtun · 2026-10-16
- Opus 5.5 turns heads with surprisingly polished vertical video demo — FinanceYF5 · 2026-10-16