Dev Critiques ARC-AGI-3: Already Saturated, Quadratic Scoring Inflates Egos
mgostIH · x · 2026-07-30
A developer points out that the ARC-AGI-3 benchmark is already saturated by current testing harnesses, meaning even minor reinforcement learning (RL) can massively inflate model scores.
Furthermore, the author heavily criticizes the benchmark's scoring system. They argue that calculating the score using a quadratic percentage of solved tasks serves no purpose other than to artificially boost certain egos, especially when early scores were all near 0%.
Related event: ARC-AGI 3 Evaluation Mechanism Under Fire from Developers(9 posts)→
More from Research
- Transforming quadruped robot runs on real hardware: per-pose RL policies in Isaac switched on morph — _Stocko_ · 2026-10-16
- KinoGen synthesizes robot loco-manipulation reference motions directly via kinematically grounded video generation — ChongZzZhang · 2026-10-16
- Science Robotics paper introduces F³, predictive event-based vision beyond optical flow — CSProfKGD · 2026-10-16
- VideoGen-Agent: An 8B Multimodal Agent Turns Video Generation Into a Tool-Use Loop — jiqizhixin · 2026-10-16
- Comprehension Audits: New Paper Proposes a Gate for AI-Written Code at Frontier Labs — ronbodkin · 2026-10-16
- Not Boring #215: Cammarata's Clean Networks Claim, Tesla's Flying Roadster, Tau Cut 73% — Not Boring (Packy McCormick) · 2026-10-16