Even TC0-safe state-tracking tasks: Transformers generalize worse than linear RNNs
jowenpetty · x · 2026-11-04
Allen AI researchers concede Transformers are contained in TC0 (or stricter depending on the attention model), but the paper's key takeaway is that even for state-tracking tasks solidly inside TC0, Transformers show worse generalization than linear RNNs. Accompanying figures map architectures onto the state-tracking complexity hierarchy (AC0/TC0/NC1) and show GDNs generalize perfectly on r-trivial, aperiodic and solvable data, while Transformers fail beyond their training distribution and even struggle in-distribution on general finite data.
More from Research
- RSNA 2026 Deep Learning Lab to teach medical imaging VLM dataset building from scratch — PRouzrokh · 2026-11-04
- RSNA lab details: build medical VLM datasets with EHR cohorts, de-ID and QC in Colab — PRouzrokh · 2026-11-04
- OpenAI finds agents score worse on safety evals than production; eval-objective awareness drives the gap — kaicathyc · 2026-11-04
- ByteDance Seed's Recycler recycles hidden states for near-free latent reasoning — JohnCLangford · 2026-11-04
- Direct weight manipulation beats fine-tuning: pi0.5 hits 75% on cloth-folding — ZeeshanZiaML · 2026-11-04
- Researchers propose 'Four As' framework to untangle AI governance trade-offs — Brighter-Side-News · 2026-11-04