Astra's 98.6% ARC-AGI-3 score called misleading: fair comparison shows just 54.8%
PsychologicalSoup251 · reddit · 2026-09-04
A Reddit analysis argues OpenAI's reported 98.6% ARC-AGI-3 score for Astra is technically true but deliberately misleading:
- Uneven conditions: Astra used a custom agentic harness at MAX thinking, while GPT 5.6 Sol (7.8%) and Claude Opus 5 (30.2%) ran standard harnesses, with Opus 5 only at HIGH thinking.
- Saturation: Under the same custom-harness rules, older GPT 5.6 Sol has already scored 99.0–99.9%, and one harness pushed both Sol and Opus 5 to 100.0%.
- Fair comparison: On the standard harness at HIGH thinking, Astra scores 54.8% vs Opus 5's 30.2% — a real but far smaller lead.
The author concludes ARC-AGI-3 is oversaturated and fundamentally harness-dependent, useful only for measuring LLM+harness combos rather than raw model ability.
Related event: OpenAI Launches GPT-6 Astra, Declaring the AGI Era(388 posts)→
More from Models
- Mistral's Voxtral 3 speech models land #3 on STT leaderboard at $13/1M chars — ArtificialAnlys · 2026-10-21
- Perplexity ships Decider V1.2, tops Vals AI's new ClaimBench decision-model benchmark — bo_wangbo · 2026-10-21
- Omnii-1 genome model debuts: first AI-designed phages ever given to a human patient — JosephJacks_ · 2026-10-21
- Nous Portal Pricing Revealed: Unused Credits Roll Over, Up to $100 Cap on $200 Tier — alexcovo_eth · 2026-10-21
- 1.3B open-source doc parser OXR tops three benchmarks, parsing 100 pages in 101 seconds — s_mohinii · 2026-10-21
- How LLMs store facts: three learning phases, attention circuits, and hallucinations — scott_linderman · 2026-10-21