AI2's StateBench: frontier models like GPT-6 Sol drop to 13% on long-text state tracking
allen_ai · x · 2026-11-04
AI2 researchers released StateBench, a new eval testing whether LLMs can track state changes over long texts (object handoffs, variable reassignments). Frontier models including GPT-6 Sol and Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems.
More from Models
- '6 Astra High' spotted in model picker, hinting at new Astra release — gethackteam · 2026-11-04
- DeepSeek V4.1 Flash reads raw weights: 3 of 4 RL checkpoints identified correctly — poppear · 2026-11-04
- Blogger: Everyone's Sleeping on Google's Latest Models — 'They're Back' — iruletheworldmo · 2026-11-04
- Blogger Claims OpenAI Astra 6.2 Cuts Price 60% to Match Opus 5.5 — bindureddy · 2026-11-04
- Sarvam Prism launch demoed with a Snake game comparing decision models vs LLMs — itsOmSarraf_ · 2026-11-04
- Aohan: inference should be fast enough that the environment becomes the bottleneck — _arohan_ · 2026-11-04