TextNCA Tests Neural Cellular Automata for Language Modeling, Trails Transformers
burny_tech · x · 2026-11-10
An EMNLP 2026 Findings paper, TextNCA, ports the neural cellular automata (NCA) principle from vision to language modeling.
- Architecture: token positions act as cells in a 1D causal model, windowed attention gives local perception, and a GRU-style gate updates states in place; a hierarchical variant cascades widening attention windows w ∈ {8,32,128}, reusing each stage's parameters for Ts iterations.
- Results: 30M-parameter models trained on SlimPajama for 60k steps reach 60.3 best PPL on WikiText-103 vs 52.8 for a similar-size 6-layer Transformer — an analytical probe, not a performance alternative.
- Key finding: reordering or flattening the windows raises PPL by 16.7–70.8, while a non-iterating sliding-window Transformer preserving the staged order performs comparably — the narrow-to-wide schedule is the dominant factor; iterated weight sharing shows no reliable benefit at this scale.
- Intermediate-state analyses support a prepare-then-decode account: early stages reorganize representations, the final wide stage does most next-token loss reduction.
More from Research
- WVU researcher presents 10 years of tractometry work on Alzheimer's white matter vulnerability — PTenigma · 2026-11-10
- New cryo-EM structure: pSTAT1 bound to importin alpha 5 (PDB 11GJ) — chaitjo · 2026-11-10
- ARLoop: fully-open 1.2B looped transformer beats Huginn-3.5B and Qwen3.5-2B on all 8 benchmarks — hhsun1 · 2026-11-10
- RAGCap-Bench: A Fine-Grained Benchmark for Intermediate Capabilities in Agentic RAG — kalyan_kpl · 2026-11-10
- RBA gives LLaVA-OneVision-2 a 3x lossless training speed-up, merged upstream — liuziwei7 · 2026-11-10
- Networks aren't scale invariant — so why has scale invariance motivated so many useful methods? — jasondeanlee · 2026-11-10