Allen AI's StateBench under fire: critics question oracle-max metrics on stochastic benchmark
ahatamiz1 · x · 2026-11-04
AI2 (Allen Institute for AI) released StateBench and reported that LLMs trained directly on it, then evaluated on longer inputs, show hybrids and Gated DeltaNets near-perfect on the first 3 levels while transformers degrade with length and can't solve general finite tasks.
Users then publicly questioned the evaluation methodology: whether oracle-max metrics were used (any 1 of N runs solving counts as success), and whether baselines got proper initialization, with one critic arguing that's the wrong way to report such a stochastic benchmark — and that AI2 "used to be a credible name."
- lambdaviking responded that the benchmark is fully deterministic (ignoring hardware nondeterminism), so no oracle-max/pass@k style metrics were reported.
- jowenpetty pressed for the exact definition of oracle-max, which ahatamiz1 explained as counting any single successful run out of N.
More from Models
- DeepSeek quietly launches "DeepSeek Harness" YouTube channel, hinting at plugin-based agent framework — gnukeith · 2026-11-04
- Reddit user seeks uncensored local model that runs on a weak PC for search and bulk downloads — Emotional_Orange_279 · 2026-11-04
- DeepSeek partial outage fuels speculation compute is shifting to unreleased V4.1 Pro — AccBalanced · 2026-11-04
- Claude hand-writes a 984-param neural net for C syntax highlighting, 5x faster than Pygments — AlpinDale · 2026-11-04
- Fireworks Forge keynote: Sovereign stack, Ember next-gen targeting 30x efficiency, No-GPU — AccBalanced · 2026-11-04
- Kimi K3.1 Pro weights drop tomorrow: 2.8T params, 1M-token context — togethercompute · 2026-11-04