Ai2 releases StateBench: Gated DeltaNet tracks state over sequences 10× longer than training
allen_ai · x · 2026-11-04
Ai2 introduced StateBench, a benchmark testing how well language models track entity state across long documents—from variable assignments in code to objects changing hands in stories. Key findings:
- On the hardest level, Gated DeltaNet handled sequences 10× longer than those seen in training, thanks to its recurrent running memory.
- A new proof shows Gated DeltaNet with non-negative eigenvalues can solve some periodic tasks under fixed-precision arithmetic with rounding.
- The benchmark grades state-tracking by dependency-chain complexity, exposing architectural differences that hardest-only tests miss; Transformers have known theoretical limits here.
Dataset and blog are publicly available for researchers to test new models and extend state-tracking data.
More from Research
- Researchers Push Back: Frontier Labs Are Overlooking Chemistry in AI for Science — anshulkundaje · 2026-11-04
- Coding agents build baselines for 36 top-journal AI methods, matching ~95% of scores — owl_posting · 2026-11-04
- EGA beats PCA at recovering dimensions from LLM item embeddings, study finds — GolinoHudson · 2026-11-04
- Speculative decoding math explained: acceptance rate equals 1 minus TV distance — techNmak · 2026-11-04
- AgentTime Adds Combined Time-Awareness Score; GPT Models Clearly Ahead — maksym_andr · 2026-11-04
- Researchers build Pixel: a co-designed full-stack system for mobile manipulation — kevin_zakka · 2026-11-04