Coding Agent's ARC-AGI-3 100% Claim Rebutted by Chollet
A general-purpose coding agent claimed a 100% score on ARC-AGI-3, but benchmark author François Chollet publicly rebutted the claim, arguing that a model never formally evaluated on the private benchmark should not claim full marks. The dispute centers on the distinction between the public evaluation set and the private test set, as well as the lack of a semi-private scoring channel for frontier model APIs this year due to Kaggle compute limits.
Confirmed
- Citing the ARC-AGI-3 paper released in March, Chollet noted that the public evaluation set was designed as a demo environment "easy for humans and AI, prioritizing clarity and fun," and does not represent the formal private test set, so passing the public set alone does not equal a full score on the benchmark.
- Chollet explained the evaluation pipeline: the ARC 3 competition runs on Kaggle with a private dataset; a separate pipeline runs for frontier model APIs upon partner request.
- Responding to complaints about limited Kaggle compute, Chollet countered that a team achieved comparable results on Kaggle last year, and that the Kaggle competition only runs private-set testing.
- JFPuget emphasized they only tested on public evaluation games, and that the lack of tooling for semi-private data evaluation is why they could not submit to the community leaderboard.
- JFPuget added that due to compute limits, Kaggle did not provide semi-private scoring runs for frontier model APIs this year as it did for ARC-AGI 2, leaving frontier models without a fair channel to participate.
Why it matters
- The dispute touches the core of benchmark credibility: results on a public demo set must not be conflated with private test-set scores, and claiming unearned full marks misleads the community's judgment of model capability.
- The missing semi-private scoring channel means frontier models and the community leaderboard lack a unified comparable channel this year, weakening the comparability of the evaluation ecosystem.
2026-08-31 ~ 2026-09-01 · 6 related posts
Primary sources
- [source] François Chollet explains ARC-3 evaluation on Kaggle — fchollet · 2026-08-31
- ARC-AGI 3 controversy: Kaggle limits prevent frontier model API evaluation — JFPuget · 2026-08-31
- Chollet responds to ARC-AGI eval dispute: don't claim untested scores — fchollet · 2026-08-31
- [source] Dispute: Claiming 100% on benchmark without private eval is invalid — JFPuget · 2026-08-31
- [source] Chollet debunks "100% on ARC-AGI-3" claim: only on easy public set — fchollet · 2026-09-01
- ARC-AGI-3 perfect score challenged: public set was designed to be easier — JFPuget · 2026-09-01