Chollet responds to ARC-AGI benchmark scoring dispute

François Chollet responded to ARC-AGI disputes, saying models never evaluated on a benchmark should not claim 100%, while JF Puget noted Kaggle's compute limits prevented semi-private API scoring runs for ARC-AGI 3, making fair leaderboard participation impossible.

2026-08-31 ~ 2026-08-31 · 4 related posts

• François Chollet explains ARC-3 evaluation on Kaggle — fchollet · 2026-08-31
• ARC-AGI 3 controversy: Kaggle limits prevent frontier model API evaluation — JFPuget · 2026-08-31
• Chollet responds to ARC-AGI eval dispute: don't claim untested scores — fchollet · 2026-08-31
• Dispute: Claiming 100% on benchmark without private eval is invalid — JFPuget · 2026-08-31