AI2's StateBench: frontier models like GPT-6 Sol fall to 13% on long-context state tracking
OfirPress · x · 2026-11-04
Allen AI released StateBench, a benchmark testing whether LLMs can track state changes over long texts (object handoffs, variable reassignments). On the hardest, longest problems, frontier models including GPT-6 Sol and Claude Opus 5.5 fell to 13% accuracy or lower. OfirPress's team is already using it internally to develop new Olmo models.
More from Models
- DeepSeek quietly launches "DeepSeek Harness" YouTube channel, hinting at plugin-based agent framework — gnukeith · 2026-11-04
- Reddit user seeks uncensored local model that runs on a weak PC for search and bulk downloads — Emotional_Orange_279 · 2026-11-04
- DeepSeek partial outage fuels speculation compute is shifting to unreleased V4.1 Pro — AccBalanced · 2026-11-04
- Claude hand-writes a 984-param neural net for C syntax highlighting, 5x faster than Pygments — AlpinDale · 2026-11-04
- Fireworks Forge keynote: Sovereign stack, Ember next-gen targeting 30x efficiency, No-GPU — AccBalanced · 2026-11-04
- Kimi K3.1 Pro weights drop tomorrow: 2.8T params, 1M-token context — togethercompute · 2026-11-04