Hashimoto's Stanford CS312 quiz: what happens if you untie a looped Transformer mid-training?

tatsu_hashimoto · x · 2026-11-19

Tatsu Hashimoto posted a CS312 (Stanford, Fall 2026) deep-learning intuition quiz on Looped Transformers: models that reuse one middle block six times to save parameters — but what if you undo the weight sharing halfway through training?

Three runs are compared: Looped (same block reused throughout), Untied midway (clone the shared block into six independent trainable copies at the halfway point), and Dense (six independent blocks from initialization). All execute eight blocks on the same 614.4M-token budget, peak LR 0.003, with Adam state reset midway. The question: which achieves the lowest final validation loss — and does untied midway end up closer to dense or looped?

Related event: Stanford Quiz Shows Looped Transformers Benefit from Untying Weights Midway(2 posts)→

Original post →

More from Research

Research channel →