Chollet: ARC 3's public games are a demo set, not an eval; Kaggle leader sits at 2.70%
fchollet · x · 2026-08-14
François Chollet, founder of ARC Prize, is reiterating that ARC 3's public games are a demonstration set — not a training set and not an eval. It is not meant to be used as training data or for evaluation, and scores on the public demonstration set are not indicative of scores on the actual benchmark; its purpose is to demonstrate the format and drive human engagement.
The private eval set is substantially more difficult and more novel. He notes that the top Kaggle leaderboard score today is 2.70% — and that is on the semi-private set; at the end of the competition, submissions will be scored on the fully private set.
More from Research
- Prime Flash MoE: Blackwell-Optimized CUDA Kernels Speed Up MoE Inference by 2.4x — pbaylies · 2026-08-14
- Why MiniMax Music3 needs an RVQ tokenizer: a technical explainer — ostrisai · 2026-08-14
- Philosopher Chalmers analyzes Anthropic's J-space: not a global workspace — burny_tech · 2026-08-14
- Chinese chemogenetics trial not on ClinicalTrials.gov; watch ChiCTR for FIH trials — MWCvitkovic · 2026-08-14
- Universal hand action space enables cross-embodiment robot hand skill learning — chris_j_paxton · 2026-08-14
- Fisher-Rao Distance Between Multivariate Normals Can Be Approximated Arbitrarily Finely — FrnkNlsn · 2026-08-14