Benchmark errors found in CritPt; GPT-5.6 hits 94.4% pass@4 after fixes
bookwormengr · x · 2026-09-15
The CritPt physics-reasoning benchmark originally had top scores starting around 9–13% and plateauing near 30–32% across model releases. A new audit found errors in 21 of 56 questions. After repairing or removing them, GPT-5.6 Sol reached 94.4% pass@4 on the corrected benchmark. Though not directly comparable to the official leaderboard, this strongly suggests the physics–math reasoning gap was overestimated — frontier models are likely much better at physics reasoning than previously thought.
More from Models
- Claude upgrade-page traffic rebounded to 17.6M in September, near May's 18.6M peak — firstadopter · 2026-11-02
- Open-source GLiNER-PII scrubs 13 identifiers from a discharge note in 550ms on GPU — MaziyarPanahi · 2026-11-02
- OpenAI Dots user says bloat and broken infra drove him to try newcomer Muse — Koala_Confused · 2026-11-02
- Fable 5.5 reportedly one-shots animation in pure code as Anthropic's coding bet pays off — Hesamation · 2026-11-02
- One agent goal has been running for 4 days, burning 30K tokens — CtrlAltDwayne · 2026-11-02
- Daily AI Brief: Gemini 4 Carbon spotted on Arena, DeepSeek open-sources coding agent — testingcatalog · 2026-11-02