MIT & Adobe's Latent Lattice LMs cut perplexity 15.41→14.40, matching 50% larger Transformers
burny_tech · x · 2026-11-19
Researchers from MIT and Adobe (Shivam Duggal, William Freeman, Antonio Torralba, Richard Zhang, Eli Shechtman, et al.) introduce Latent Lattice Language Models (LLLMs), a new axis for scaling LM computation: increasing the number of active computational states rather than depth or looped layers.
- Each Transformer layer gets per-layer latent streams that recur along time alongside the vertical residual stream, conditioning attention, KV-cache writes, MLP computation, and decoding — unlike Transformer-SSM hybrids where recurrent states mainly serve sequence mixing.
- Trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 vs. a parameter-matched Transformer, matching a conventional model with 50% more parameters.
- It composes with prior residual-stream enhancements (hyper-connections, input residuals), pushing perplexity down to 13.61.
- Gains extend to Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids, with Lattice variants beating pure baselines with 4x larger native states.
Takeaway: persistent per-layer state is a complementary scaling axis for language-model compute.
More from Models
- Miles Brundage: GPT-6.2 Astra appears to have regressed on long palindromes — Miles_Brundage · 2026-11-19
- Opus 5.5 botches a simple experiment: picks a constant-token task, then blames the API — adam_karvonen · 2026-11-19
- Anthropic accused of banning AI safety researcher across multiple Ultra subscriptions — Sauers_ · 2026-11-19
- Tencent Hunyuan's Hy4-Flash engine PR reveals tensor parallel, MTP drafts, CPU expert lane — TencentHunyuan · 2026-11-19
- Hunyuan confirms Hy4-Flash is an unexpected hit, free on OpenCode 10 more days — TencentHunyuan · 2026-11-19
- Telling Gemini 4 Carbon it's being benchmarked bypasses its 'contentious topic' refusals — panickssery · 2026-11-19