Ai2 Releases StateBench: Frontier Models Drop Below 13% on Long-Context State Tracking

The Allen Institute for AI (Ai2) has released StateBench, a new benchmark that systematically tests LLMs' ability to track state changes over long texts—such as items changing hands, variables being reassigned, and following a ball hidden under cups that are repeatedly shuffled. The results show that frontier models, including GPT-6 Sol and Claude Opus 5.5, drop below 13% accuracy on the hardest, longest problems, revealing that state tracking over long tasks remains a significant weakness of current models.

Confirmed

• StateBench categorizes state tracking into four difficulty levels: r-trivial (recalling fixed information like birthdays), aperiodic, solvable, and general finite (e.g., continuously tracking a ball under shuffled cups), providing a difficulty gradient for pinpointing model failures.
• At the hardest level, frontier models (including GPT-6 Sol and Claude Opus 5.5) fall below 13% accuracy.
• To rule out the possibility that failures stem from pretraining rather than architecture, Ai2 trained models directly on StateBench and evaluated them on longer inputs: Hybrid architectures and Gated DeltaNet approach near-perfect scores on the first three difficulty levels, while Transformers degrade with length.
• Gated DeltaNet can handle sequences roughly 10x longer than those seen during training; the team also provides a new proof showing that Gated DeltaNet with non-negative eigenvalues can still solve a subset of problems under fixed-point arithmetic with rounding.

Why it matters

• State tracking is a foundational capability for reading code (variable assignment) and understanding stories (items changing hands, characters moving), and this benchmark offers a graded tool for measuring it.
• The controlled experiments suggest the failures may lie more in the Transformer architecture itself than in training data, and the length-extrapolation advantages of linear/hybrid architectures like Gated DeltaNet offer new evidence for designing long-context models.

2026-11-04 ~ 2026-11-04 · 7 related posts

• Can LLMs track state over long texts? AI2 opens StateBench, open-sourced on GitHub — allen_ai · 2026-11-04
• StateBench's four difficulty levels, from remembering a birthday to tracking swapped cups — allen_ai · 2026-11-04
• [source] Ai2's StateBench: frontier models drop to 13% accuracy on long state-tracking tasks — allen_ai · 2026-11-04
• [source] Trained directly on StateBench, hybrids and Gated DeltaNet ace state tracking where transformers fail — allen_ai · 2026-11-04
• [source] Gated DeltaNet generalizes ~10× beyond training lengths on StateBench's hardest level — allen_ai · 2026-11-04
• Ai2 releases StateBench: Gated DeltaNet tracks state over sequences 10× longer than training — allen_ai · 2026-11-04
• AI2's StateBench: frontier models like GPT-6 Sol drop to 13% on long-text state tracking — allen_ai · 2026-11-04