First PhD paper accepted at NeurIPS 2026: Sparse layers key to scaling looped LMs
burny_tech · x · 2026-09-27
USC PhD student Ryan Lee's first paper, "Sparse Layers are Critical to Scaling Looped Language Models," is accepted at NeurIPS 2026. The paper compares standard transformers, MoE models, and looped architectures, with two main findings:
- Looped-MoE scales better than the standard baseline, while dense looped models do not. The reason is routing divergence across loops: different experts are activated on each pass through the same shared layers, recovering expressivity without extra parameters.
- Loop boundaries are superior early-exit points: since each loop ends with the layers that produce the final output, early exits at loop boundaries yield better compute-quality trade-offs than in standard models.
The authors conclude that a Looped-MoE model with early exits can beat standard transformers at scale while delivering significant memory and inference savings with minimal quality loss — a step toward compute-adaptive LMs.
More from Models
- Fable 5.2 Leads MazeBench Token Efficiency; Billion-Token Runs Underway — patience_cave · 2026-11-16
- Fable 5.2 Tops MazeBench With 21% After 90 Hours and $6,000 in Credits — patience_cave · 2026-11-16
- Tencent Hy4 flash and InternLM Sage-S2.1-Flash both released, tool-calling tests requested — MajesticPuzzler · 2026-11-16
- User Shares Coral Reef Creation Labeled 'Qwen 4 Max (xhigh)', an Unannounced Model Name — intellectronica · 2026-11-16
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16