Frontier models score under 13% on long-horizon StateBench, but skeptics question benchmark validity
ahatamiz1 · x · 2026-11-04
allenai released StateBench, a benchmark testing whether LLMs can track state changes over long texts (objects changing hands, variable reassignments). Frontier models including GPT-6 Sol and Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems.
@ahatamiz1 pushes back: linking a synthetic state-tracking benchmark to real-world code and file tracking is misleading — doing well on a toy task doesn't mean a model will track real codebases better. He adds a theoretical note: fixed-depth transformers are in TC⁰, so under standard complexity assumptions they can't solve hard state-tracking problems like permutation composition in a single forward pass.
More from Models
- DeepSeek quietly launches "DeepSeek Harness" YouTube channel, hinting at plugin-based agent framework — gnukeith · 2026-11-04
- Reddit user seeks uncensored local model that runs on a weak PC for search and bulk downloads — Emotional_Orange_279 · 2026-11-04
- DeepSeek partial outage fuels speculation compute is shifting to unreleased V4.1 Pro — AccBalanced · 2026-11-04
- Claude hand-writes a 984-param neural net for C syntax highlighting, 5x faster than Pygments — AlpinDale · 2026-11-04
- Fireworks Forge keynote: Sovereign stack, Ember next-gen targeting 30x efficiency, No-GPU — AccBalanced · 2026-11-04
- Kimi K3.1 Pro weights drop tomorrow: 2.8T params, 1M-token context — togethercompute · 2026-11-04