Meta paper: looping transformer weights adds depth without more parameters, up to 32 loops
iScienceLuvr · x · 2026-10-31
A new Meta FAIR paper (arXiv:2610.40325) systematically studies looped transformers—reusing weights to deepen models without adding parameters. Experiments span up to 1B parameters, 155B tokens, 1.07×10²¹ FLOPs, and 32 loops.
- Looping the full transformer gives the best quality at a fixed parameter budget; looping a middle block wins when training FLOPs are constrained
- Gains saturate, reverse, or destabilize beyond some loop counts
- Small loop-specific low-rank adapters help increasingly as loops multiply
- Naive Parallel Looped Transformers lose the gains; preserving the current-token representation with separate normalization and controlled combination restores performance
More from Research
- 4B decision model trained on 117k examples matches Jev — except it can't detect sarcasm — SRSchmidgall · 2026-10-31
- David Baker Lab Preprint: Protein Design Cuts CD8⁺ T Cell Immunogenicity — gottapatchemall · 2026-10-31
- Companies pay $300 per paper to survey ICLR/NeurIPS authors — grads can cash in — Icy-Park4287 · 2026-10-31
- Gary Marcus mocks OpenAI's new algorithm: exponential subroutine below a 10^25-variable threshold — AlexTensor · 2026-10-31
- After NVIDIA's BindClaw, every AI4Bio paper now needs an agentic pipeline demo to prove versatility — rishabh16_ · 2026-10-31
- Neural Thickets Revisited: Frozen-Model Logit Perturbations Match Full-Weight Perturbation Ensembles — kastnerkyle · 2026-10-31