GPT-6 Astra disputed expert answers on a physics benchmark — and was right, scoring 98%
josh_bickett · x · 2026-10-15
- A model called GPT-6 Astra scored +80% on a physics benchmark and insisted the expert solutions were wrong while its own calculations were correct.
- A team audited the disputed problems and confirmed Astra was right; its score was revised up to 98%.
- An excerpt (screenshot) shows evaluators initially flagged more errors, but on closer review the original expert solutions contained subtle mistakes — after corrections, Astra's solutions held up in every case.
- Andrew Curran ties this to Elon Musk's recent remarks about models behaving erratically in RL when hitting situations like this.
More from Fun
- Vercel demoed an F1 simulator where you race against AI models — evilrabbit_ · 2026-10-15
- Vibe coding burn: game fully loads before your buy button does — TAbrodi · 2026-10-15
- Grok Bot Builds a Wall of 1,600 Real X Profile Pics Live — sierracatalina · 2026-10-15
- From throwing punches at humans to dunking on undergrad social media interns — RylanSchaeffer · 2026-10-15
- Why is my Oura ring talking to me like a therapist? — var_epsilon · 2026-10-15
- Crimsonland 0.17 adds custom quests you can build by describing ideas to an agent — banteg · 2026-10-15