ARC-AGI-4 should stay private, after Opus 5 scored 3× the next-best model on ARC-AGI-3
burny_tech · x · 2026-07-25
- The post argues that ARC-AGI-4 should probably be kept more private, with far fewer public demos.
- It suggests that many people now assume ARC-AGI progress is being driven by targeted RL environments rather than broad, public benchmark play.
- The quoted context notes that on ARC-AGI-3, a benchmark where models must solve novel problems, Opus 5 scored 3× higher than the next-best model.
- The discussion is less about a product announcement and more about how benchmark design and publicity may affect what progress looks like.
Related event: Opus 5's High ARC-AGI-3 Score Sparks Cheating and Overfitting Controversy(11 posts)→
More from Models
- repligate: noisy context is genuinely 'hard' for LLMs, beyond metaphors — repligate · 2026-10-27
- The trust paradox: only the least trustworthy LLMs will do the sketchiest things — AaronBergman18 · 2026-10-27
- Integrity and refusal may correlate: the fundamental tradeoff behind password refusals — AaronBergman18 · 2026-10-27
- Claude's refusal to type one-time passwords is a real security impediment — AaronBergman18 · 2026-10-27
- repligate argues context full of noise is genuinely 'hard' for LLMs, mulls an info-theory model of breaks — repligate · 2026-10-27
- Fork builds engine to run Qwen at ~400 tok/s on three used V100s — jfinkpottery · 2026-10-27