Chollet responds to ARC-AGI eval dispute: don't claim untested scores
fchollet · x · 2026-08-31
François Chollet responded to the controversy surrounding the ARC-AGI benchmark. He stated that one should not claim a 100% score on a benchmark if the model has never actually been evaluated on it. Regarding the claim that Kaggle lacks compute, he noted that teams achieved scores like 72.9% on the platform last year, and emphasized that there are many private benchmarks available for evaluation outside of ARC-3.
Related event: Coding Agent's ARC-AGI-3 100% Claim Rebutted by Chollet(6 posts)→
More from Research
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Nature paper uses language models to map single neurons behind human language — raphaelmilliere · 2026-10-16
- Reviewer shares workflow: feeding the finished review — never the paper — to AI for a second pass — StingMeleoron · 2026-10-16
- 3SUM record falls to O(n^1.921), Lean-verified — but will it yield usable algorithms? — iskander · 2026-10-16
- Ablation studies to help Kaggle Gemma 4 competition participants — ayushthakur0 · 2026-10-16