Stanford Quiz Shows Looped Transformers Benefit from Untying Weights Midway

Tatsu Hashimoto posed a Stanford CS312 quiz on Looped Transformers, finding that while looping one block is worse than a dense model (loss 2.9333), untying the shared weights midway recovers about 61% of the gap.

2026-11-19 ~ 2026-11-19 · 2 related posts