Can LLMs track state over long texts? AI2 opens StateBench, open-sourced on GitHub
allen_ai · x · 2026-11-04
Allen AI opens its thread: can LLMs track state changes over long texts—an object changing hands, a variable reassignment? On the new StateBench eval, frontier models including GPT-6 Sol and Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. The repo (allenai/state-bench, Apache-2.0) generates reproducible state-tracking evals from canonical assignment traces and stages Hugging Face datasets.
More from Research
- STOC submissions include dueling P=NP and P!=NP proofs; blogger bets both wrong — thegautamkamath · 2026-11-04
- DeepSeek V4.1 Flash reads raw weights: 3 of 4 RL checkpoints identified correctly — poppear · 2026-11-04
- CoworkOS Debuts: Three-Layer Agent Coworker Framework Lifts Qwen3.6 by 9.76 Points — KyeGomezB · 2026-11-04
- STOC 2027 draws 1665 submissions and publishes all of them, with author names, immediately — thegautamkamath · 2026-11-04
- DOLORES: Training-Free Meta-Scaffold Lifts 8B Models Past 32B Baselines — StellaLisy · 2026-11-04
- Formalizing PTX in Lean to formally verify GPU kernels written by AI agents — gharik · 2026-11-04