Ai2's StateBench: frontier models drop to 13% accuracy on long state-tracking tasks
allen_ai · x · 2026-11-04
Ai2 released StateBench, a benchmark that grades state tracking into 4 levels: r-trivial (recalling fixed info like a birth date), aperiodic, soluble, and general finite (e.g. tracking a ball as cups are repeatedly swapped).
It evaluated open pretrained models plus frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol + Luna. Frontier models did well on simpler problems, but accuracy dropped sharply as tasks got harder and inputs got longer—falling to 13% or lower on the longest general finite problems.
More from Models
- Ollama bets on zero data retention to make open models the enterprise default — ollama · 2026-11-04
- CMU's cua-speedrun ranks computer-use agents on speed and cost: GPT-6.1 Terra fastest, Opus 5.5 top score 85.1% — kohjingyu · 2026-11-04
- Astra 6.2 price rumor: 60% cheaper than 6.0, seen as OpenAI's counter to Anthropic — mark_k · 2026-11-04
- 27B Amos beats Sarvam's 105B MoE on Decision Bench, 94.9% vs 92.5% — cneuralnetwork · 2026-11-04
- Indian team claims Wity swept Sarvam 4-0 on JevBench despite Sarvam's $389M raise — cneuralnetwork · 2026-11-04
- Crosby launches Surgeon model to judge surgical contract redlines — graceisford · 2026-11-04