ARC-AGI-4 to benchmark open-ended innovation; Kenneth Stanley on why that's nearly impossible
tobias_rees · x · 2026-09-13
ARC Prize announces ARC-AGI-4, an open-source benchmark for autonomous open-ended innovation, arguing humans still far outperform AI at open-ended invention — the meta-skill for progress in every field. Reposter Kenneth Stanley endorses the direction but flags the core tension: openness is fundamentally resistant to benchmarking, which is why the field has lagged. He reviews past attempts, from Bedau's activity statistics in artificial life to ANNECS (Accumulated Number of Novel Environments Created and Solved) from his Enhanced POET paper, noting no benchmark yet allows seamless head-to-head comparison of arbitrary systems.
Related event: ARC-AGI-4 Benchmark to Target Autonomous Open-Ended Innovation(4 posts)→
More from Research
- AISTATS 2027 workshop proposals due Nov 6 after one-week extension — qberthet · 2026-10-31
- Stanford puts 10 full AI courses on YouTube for free, taught by Ng and Manning — stanfordnlp · 2026-10-31
- krishnan on autonomous lab agents: criticism changed the next search, not just verdicts — krishnan · 2026-10-31
- Testing agents with explicit rejection records: a cheaper path to verified troubleshooting answers — krishnan · 2026-10-31
- krishnan proposes logging ruled-out troubleshooting hypotheses for future reopening — krishnan · 2026-10-31
- Multi-agent AI lab autonomously researches quantum spins in diamond, revises drifting experiments — krishnan · 2026-10-31