ARC AGI 3 should have stayed private, with no examples or public dataset
flowersslop · x · 2026-07-27
The post argues that ARC AGI 3 should have been kept fully private: no example games, no public dataset, and no detailed public hints beyond a vague description like “1990s-style 2D games.”
The point is that once the benchmark’s contents are visible, both labs and the public can adapt to it, weakening its value as a clean test of general intelligence.
More from AGI Musings
- For many users, the real ChatGPT question is trust, not token limits — petergyang · 2026-07-27
- AI makes knowledge cheaper, but judgment remains the scarce skill — _jaydeepkarale · 2026-07-27
- A reply to François Chollet argues intelligence is only part of the advantage — binarybits · 2026-07-27
- A Chollet quote reignites the debate over whether intelligence has diminishing returns — binarybits · 2026-07-27
- Noahpinion quotes Chollet: intelligence may hit a hard ceiling — binarybits · 2026-07-27
- Using LLMs to optimize the next LLM is not singularity, argues Burkov — burkov · 2026-07-27