Looped Transformers explained: same weights, evolving hidden states, extra compute per parameter
TheTuringPost · x · 2026-11-17
Turing Post breaks down Looped Transformers: the same Transformer block (or stack) runs for multiple passes, feeding each output back as input. What loops is the hidden state—token embeddings initialize it, the shared stack updates it via attention and FFN, and the output head converts the final representation to token scores. The companion chart contrasts 24 unique blocks × 1 pass vs 6 blocks × 4 passes (both = 24 block applications): same effective depth without per-pass weights, with fixed or adaptive stopping.
More from Research
- PyTorch finally ships stateless RNG API after years of userland workarounds — ezyang · 2026-11-17
- Why tensor parallelism breaks RNG reproducibility in model init — ezyang · 2026-11-17
- Philox RNG-to-tensor assignment is extremely sensitive to hardware details — ezyang · 2026-11-17
- Authors say they reproduced their ICLR 2027 reviewer's erratic reviews with a simple GPT prompt — anand_bhattad · 2026-11-17
- Researcher's pet peeves: the clichés that plague nearly every AI paper's intro — StephenLCasper · 2026-11-17
- MIT CSAIL's TangiMap uses an AI agent to map software onto physical controls — MIT_CSAIL · 2026-11-17