OpenAI's Astra Scored 62.7% and 99.9% on the Same Benchmark, 37 Points Apart
mixtapedmonk · reddit · 2026-09-14
Digging into ARC Prize's actual results table, the author found OpenAI's GPT-6 Astra posted 62.7% and 99.9% on the same ARC-AGI-3 benchmark depending on which harness was used—37 points apart—and the org that built the test won't call it AGI. Fortune separately found five numbers quietly changed on OpenAI's own launch page after it went live. Comparing the Llama 4 benchmark precedent, the writeup concludes it could be genuine harness noise or something else, but headline-screenshot benchmark reporting is clearly broken.
More from Models
- Gemini 4 tops public coding benchmarks but drops to bottom on private tasks — Hesamation · 2026-10-30
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- EachLabs launches Lockstep recast model claiming 75% cost cut and better quality — miilesus · 2026-10-30
- ChatGPT cited a 300+ page government legal PDF it retrieved on its own — lilyraynyc · 2026-10-30
- Best MLX model for French-to-English translation on an M2 Mac? — greenreddits · 2026-10-30
- AI-driven cyber ops near the sophistication peak set by GTG-1002's Claude Code campaign — Jsevillamol · 2026-10-30