Towards Looped Models Done Right: sandwich architecture improves reasoning, transfers to MoE
IFM · hf · 2026-10-19
A team from IFM, USC, and CMU (including Eric Xing) releases part one of a systematic study on looped language models. Established architectures entangle three design axes—recurrence placement, input injection, and recurrent-state organization. The authors train models from scratch at matched parameter scale, logical depth, and token budgets to isolate each axis.
Key findings: a prelude–recurrent block–coda sandwich architecture consistently improves context extraction and reasoning-intensive tasks; input injection primarily enhances knowledge retrieval and context utilization; and the design principles transfer to resident- and active-parameter-matched Mixture-of-Experts models. Code: github.com/ifm-ai/xllm-loop.
More from Research
- 1.9 MB extracted (not trained) LoRA fixes vertical color stripes in MOSS-Image SFT — Nynxz · 2026-10-19
- Three ways to execute a AND b AND c: DataFusion filter PR explained — Abhishekcur · 2026-10-19
- ACT3D trains VLMs to actively explore 3D worlds with RL, hitting 53.2% accuracy — UCSanDiego · 2026-10-19
- OPMD: ShanghaiTech distills teacher intermediate computations beyond output supervision — ShanghaiTech · 2026-10-19
- LexFrame: interpreting LM residual streams in the model's own semantics — Jian Gu · 2026-10-19
- Amazon Paper: QA Scores Miss Agent Memory Failures by 16.8 Points, New FEB-30 Benchmark Fixes It — rohanpaul_ai · 2026-10-19