Even within TC0, transformers generalize worse than linear RNNs on state tracking
ahatamiz1 · x · 2026-11-04
Responding to pushback that transformers' inability to state-track is well-known and unrelated to language modeling, the allenai-affiliated authors clarify: while transformers are indeed contained in TC0 (or stricter under some attention models), their key finding is that even for state-tracking tasks solidly inside TC0, transformers show worse generalization than linear RNNs — with empirical plots attached. Relevant to why hybrid architectures like OLMo Hybrid include linear attention layers.
More from Models
- DeepSeek quietly launches "DeepSeek Harness" YouTube channel, hinting at plugin-based agent framework — gnukeith · 2026-11-04
- Reddit user seeks uncensored local model that runs on a weak PC for search and bulk downloads — Emotional_Orange_279 · 2026-11-04
- DeepSeek partial outage fuels speculation compute is shifting to unreleased V4.1 Pro — AccBalanced · 2026-11-04
- Claude hand-writes a 984-param neural net for C syntax highlighting, 5x faster than Pygments — AlpinDale · 2026-11-04
- Fireworks Forge keynote: Sovereign stack, Ember next-gen targeting 30x efficiency, No-GPU — AccBalanced · 2026-11-04
- Kimi K3.1 Pro weights drop tomorrow: 2.8T params, 1M-token context — togethercompute · 2026-11-04