SSMs' Expressive Power Limited: Mamba-style Models Fail at State Tracking, Paper Shows
davidbau · x · 2026-08-21
In a discussion on AI architectures, davidbau cites a paper by Merrill et al., 'The Illusion of State in State-Space Models,' which shows that state-space models (SSMs) have similar expressive limitations to transformers, confined to TC0, and cannot solve simple state-tracking problems like permutation composition, thus failing to track chess moves, evaluate code, or track entities in long narratives. Experiments confirm Mamba-style SSMs struggle with state tracking. DeltaNet is mentioned as a potential alternative.
More from Models
- AI News Digest: DeepSeek Weekend Discounts, GPT-5.6 Sol Price Cut, Alibaba's $10B AI Raise — APPSO · 2026-08-24
- Stealth Model 'Ox Alpha' Matches GPT-5.6 in Context Arena Tests — scaling01 · 2026-08-24
- Fable 5 Makes Up Only 6% of Anthropic Sales — rohanpaul_ai · 2026-08-24
- Opus 5 writes like Sorkin? How to fix wordy style — thk_ · 2026-08-24
- 2026 Chinese Model Landscape: DeepSeek V4, Kimi K3, and More Listed — TheTuringPost · 2026-08-24
- Rumor: Claude Opus 6 Leaked Specs Include 2.5M Context and Lower Costs — iamaliveix · 2026-08-24