Ai2's StateBench: frontier models handle overwrites but drop to 13% on dependent state updates
ombre-ai · reddit · 2026-11-05
Ai2 released StateBench (Nov 3), a synthetic benchmark testing whether models can compute every variable's final value after reading a long list of changes.
- Frontier models (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol/Luna) do well when the latest assignment wins.
- When updates build on each other—running totals or values derived from other variables' current values—accuracy falls sharply as inputs grow, to 13% or lower on the longest, hardest problems.
- The poster's builder reading: agent state mixes four kinds of change (static config, overwrites, running totals, derived values), and the latter two are exactly where replaying a transcript fails.
- Suggested fix (untested): keep an explicit state record outside the transcript with current values, replacement history, and dependson edges, and recompute totals in code rather than asking the model to add up history. The benchmark doesn't test memory/retrieval/tools, so this remains a hypothesis.
- Ends with community questions on how people actually store agent state and where they've hit stale-state bugs.
More from coding & agent
- Auto-unsubscribe agent built on Jev: browser-aware email triage cleans inbox in a week — threepointone · 2026-11-05
- Olki ditches LLM-based agent monitoring for strict type systems and effects algebra — bigdata · 2026-11-05
- Alpha School Launches Founders School: Graduate With a Business Doing $1M — nateliason · 2026-11-05
- Why Coda's formula language beats Notion for deterministic agent memory — bfrench · 2026-11-05
- Dev quits 11-year career, vibe-codes AAA-looking sci-fi FPS with AI in 24h, lands Gamescom booth — Aiden_Tech_Ai · 2026-11-05
- JetBrains argues IDEs still matter in the agentic coding era: 'a cockpit, not a typing tool' — Sam_Witteveen · 2026-11-05