ARC-AGI 3 controversy: Kaggle limits prevent frontier model API evaluation
JFPuget · x · 2026-08-31
JF Puget criticized the ARC-AGI 3 setup, noting that Kaggle's limited compute prevented running frontier models (unlike last year's ARC-AGI 2), making private scores impossible. François Chollet clarified that the Kaggle competition is strictly for the private set, while the separate evaluation pipeline for frontier model APIs (run upon partner request) still exists but was not utilized for this public competition phase.
Related event: Chollet responds to ARC-AGI benchmark scoring dispute(4 posts)→
More from Companies & People
- Coinbase CEO and TapTap founder donate $1M each, Omacom Foundation hits $12M — mitsuhiko · 2026-08-31
- AI Exposes GTM Headcount as Overhead, Enabling Solo Operators — Informal-Smoke2577 · 2026-08-31
- YC applications grew 60% longer in 3 years as AI writing spreads the word 'wedge' to 20% of pitches — jfiance · 2026-08-31
- "YC has fallen" went from taboo to mainstream as morale collapses — FangYi11101 · 2026-08-31
- Linear Head of Product joins OpenAI to work on Codex and ChatGPT — himanshustwts · 2026-08-31
- Neural computation researcher pivoting to industry seeks company recommendations — KordingLab · 2026-08-31