ARC-AGI 3 controversy: Kaggle limits prevent frontier model API evaluation
JFPuget · x · 2026-08-31
JF Puget criticized the ARC-AGI 3 setup, noting that Kaggle's limited compute prevented running frontier models (unlike last year's ARC-AGI 2), making private scores impossible. François Chollet clarified that the Kaggle competition is strictly for the private set, while the separate evaluation pipeline for frontier model APIs (run upon partner request) still exists but was not utilized for this public competition phase.
Related event: Coding Agent's ARC-AGI-3 100% Claim Rebutted by Chollet(6 posts)→
More from Companies & People
- Inside OpenAI's Codex Comeback: From Failed Launch to the App That Swallowed ChatGPT — every · 2026-10-16
- React veteran Eli White joins Meta Superintelligence Labs after a decade on React — Vjeux · 2026-10-16
- How the latest AI model jump is disrupting SaaS and game studio business models — The AI Daily Brief · 2026-10-16
- Joanna Shields named to 2026 Responsible Tech Guide, warns of AI adaptation gap — joannashields · 2026-10-16
- Inference startups repeat an old pattern: hype raises money, honest engineers win — gharik · 2026-10-16
- New Yorker detail: Brockman kept asking 'how do we win?' about AI safety plans — ben_j_todd · 2026-10-16