Looped LMs could use 3x fewer parameters and a 3x smaller KV cache, matching standard Transformers
ChengleiSi · x · 2026-10-03
A research thread titled "Rethinking at Fixed Points" argues looped LMs don't have to pay for every loop: they can match a standard Transformer with 3x fewer parameters and a 3x smaller KV cache, scaling by adding FLOPs at constant memory.
The key is the fixed point of the loop. The authors introduce:
- Fixed-Point Shortcuts: what fixed points buy a looped model in training (pre-training, post-training) and inference (prefill, decoding)
- A learned depth prior and orthogonal injection: two ways to shape those fixed points
A long thread follows with full details.
More from Research
- LEV: Looped decision model lifts 2.6B accuracy 63.2% to 76.2% with adaptive compute — jiqizhixin · 2026-11-19
- Aurora2-400M-A20M: a sparse MoE pretrained from scratch in 36 hours on one RTX Pro 6000 — Tall_Abrocoma_3533 · 2026-11-19
- Researcher calls METR/Anthropic '2x speedup' RSI analysis statistically dubious — jessi_cata · 2026-11-19
- Linear Probes Could Reveal Which Persona a Base Model Is Playing, Researchers Suggest — cephaloform · 2026-11-19
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19
- OpenVax codebase grows from 78K to 565K lines in 15 months — iskander · 2026-11-19