Hashimoto's Stanford CS312 quiz: what happens if you untie a looped Transformer mid-training?
tatsu_hashimoto · x · 2026-11-19
Tatsu Hashimoto posted a CS312 (Stanford, Fall 2026) deep-learning intuition quiz on Looped Transformers: models that reuse one middle block six times to save parameters — but what if you undo the weight sharing halfway through training?
Three runs are compared: Looped (same block reused throughout), Untied midway (clone the shared block into six independent trainable copies at the halfway point), and Dense (six independent blocks from initialization). All execute eight blocks on the same 614.4M-token budget, peak LR 0.003, with Adam state reset midway. The question: which achieves the lowest final validation loss — and does untied midway end up closer to dense or looped?
Related event: Stanford Quiz Shows Looped Transformers Benefit from Untying Weights Midway(2 posts)→
More from Research
- Boltz raises $110M Series A for AI biomolecular design, already working with 6 top pharma cos — GabriCorso · 2026-11-19
- LMArena overhauls agent leaderboard: Agent Arena Index scores 2.9M sessions with anchored 1000-point scale — arena · 2026-11-19
- Anatomical SOMA: simulated humans with hinge joints and human-inspired limits, demoed via 128-agent bowling — carlosdponx · 2026-11-19
- ROBOTIS fingertip tactile sensing brought into Isaac Sim as live touch heatmaps — KyleMorgenstein · 2026-11-19
- CodeGraphy v2.2 fixes AST blind spots, hits 2.6x the file recall of naive BM25 — GeologistNo4811 · 2026-11-19
- Benchtop synthesis device releases have 3x'd; half of active manufacturers are in China — JacquesThibs · 2026-11-19