ARC-AGI 3 controversy: Kaggle limits prevent frontier model API evaluation
JFPuget · x · 2026-08-31
JF Puget criticized the ARC-AGI 3 setup, noting that Kaggle's limited compute prevented running frontier models (unlike last year's ARC-AGI 2), making private scores impossible. François Chollet clarified that the Kaggle competition is strictly for the private set, while the separate evaluation pipeline for frontier model APIs (run upon partner request) still exists but was not utilized for this public competition phase.
Related event: Coding Agent's ARC-AGI-3 100% Claim Rebutted by Chollet(6 posts)→
More from Companies & People
- Engineers credited for Tesla's demo-night downforce fan and thruster integration — MatthewChang · 2026-10-16
- Team of 5 mulls dropping ChatGPT Pro for Business as OpenAI quota parity stays unclear — bettercallfool · 2026-10-16
- OpenAI's Rosalind Academic Plan offers US research labs 30 free seats for a year — The Decoder · 2026-10-16
- BotsCrew shares how company-wide Claude setup saves 2+ hours per sales proposal — max_gladysh · 2026-10-16
- Replit CEO Amjad Masad Says the Future of Work Is the 'Self-Driving Company' — TechCrunch AI · 2026-10-16
- React veteran Eli White joins Meta Superintelligence Labs after a decade on React — Vjeux · 2026-10-16