Stanford Quiz Shows Looped Transformers Benefit from Untying Weights Midway
Tatsu Hashimoto posed a Stanford CS312 quiz on Looped Transformers, finding that while looping one block is worse than a dense model (loss 2.9333), untying the shared weights midway recovers about 61% of the gap.
2026-11-19 ~ 2026-11-19 · 2 related posts
- Stanford CS312 experiment: looping transformers lose; mid-training untying recovers 61% of gap — tatsu_hashimoto · 2026-11-19
- Hashimoto's Stanford CS312 quiz: what happens if you untie a looped Transformer mid-training? — tatsu_hashimoto · 2026-11-19