WMIP Probe Shows LLM Cumulative State Tracking Depends Heavily on Output Protocols
Dengzhe Hou · hf · 2026-11-12
Researchers from Tohoku University and collaborators introduce WMIP, a working-memory-inspired programmable probe for cumulative state tracking in LLMs.
- The benchmark requires models to track 3–100 successive updates within a single prompt without writing out intermediate states, with program-computed answers and final-state, intermediate-state, or multi-target queries.
- Key finding: measured accuracy depends heavily on output protocols — existing tests like Minerva allow 4,096 output tokens, letting models write out operations and sidestep the pure state-tracking demand; different benchmarks constrain written steps differently and may not all pose this challenge.
The paper highlights that despite strong single-step arithmetic, LLMs struggle with cumulative state tracking, and prior benchmark conclusions are heavily shaped by evaluation protocol choices.
More from Models
- Tether releases BotPsy, a 0.8B phone-sized AI agent beating Qwen3.5-2B on tool use — mawcopolow · 2026-11-12
- Grok 4.8 appears to be rolling out to some Grok Bot users, matching Musk's timeline — mark_k · 2026-11-12
- Seroter's reading list: Gemini Decisions API, agent memory components, enterprise AI adoption — rseroter · 2026-11-12
- Blogger ditches Fable 5.2 for talking about him in third person; picks Opus 5.5 — PawelHuryn · 2026-11-12
- Frontier LLMs read Cell Painting images at chance level; tiny ResNet18 beats them 280x cheaper — gklambauer · 2026-11-12
- Qwen4 is coming: 'qwen4exp' commits spotted on GitHub — kinetic_energy28 · 2026-11-12