ARC-AGI Evaluation Dispute: Claude Opus Score Questioned Over API Implementation Flaw
steipete · x · 2026-07-30
Developer @steipete pointed out that Claude Opus's recent high score on the ARC-AGI evaluation might be inflated. This is due to a flaw in the model's Chat Completion API implementation, which erroneously retained reasoning tokens, an API the official documentation does not support. However, some users argue that if both Opus and GPT-5 used the same generic testing harness, the comparison remains valid, suggesting Opus generalizes better.
Related event: Claude Opus ARC-AGI Score Questioned Over API Flaw(2 posts)→
More from Models
- Reflection's Beam, a 501B-parameter open MoE model with 1M context, lands on Hugging Face under Apache 2.0 — QGallouedec · 2026-10-16
- Why are sub-15B open models so rare now? Labs seem to have settled on 24-27B minimum — Hot_Example_4456 · 2026-10-16
- User spots model openly generating "ideal responses" to train other models — liminal_bardo · 2026-10-16
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16
- Gemma 4 31B QAT on one 24GB card: 46 tok/s plain, ~80 with draft model on RTX 4090 — Subject_Poet_3861 · 2026-10-16
- JevBench v1.7.46 leaderboard: new d1-ultra tops 32 API systems at 76.1 composite — airesearch12 · 2026-10-16