Chollet: ARC 3's public games are a demo set, not an eval; Kaggle leader sits at 2.70%
fchollet · x · 2026-08-14
François Chollet, founder of ARC Prize, is reiterating that ARC 3's public games are a demonstration set — not a training set and not an eval. It is not meant to be used as training data or for evaluation, and scores on the public demonstration set are not indicative of scores on the actual benchmark; its purpose is to demonstrate the format and drive human engagement.
The private eval set is substantially more difficult and more novel. He notes that the top Kaggle leaderboard score today is 2.70% — and that is on the semi-private set; at the end of the competition, submissions will be scored on the fully private set.
More from Research
- Researcher slams papers whose stacked changes beat even weak baselines barely — retr0jirachi · 2026-10-16
- FrankenSim builds a physics-based virtual orchestra, piano and drum demos live — doodlestein · 2026-10-16
- FoveaVLA replaces wrist cameras with learned foveation, hitting 70.8% success — Haoyu_Xiong_ · 2026-10-16
- ApolloPFN: a zero-shot forecasting network that natively handles exogenous covariates — burkov · 2026-10-16
- ICLR reviewer says 4/5 papers in review pile read like AI slop — berkustun · 2026-10-16
- Tsinghua's DBS learns content-adaptive frequency band splits for audio source separation — thu-ml · 2026-10-16