Stanford CS312 experiment: looping transformers lose; mid-training untying recovers 61% of gap
tatsu_hashimoto · x · 2026-11-19
Tatsu Hashimoto published final validation losses from a CS312 quiz experiment: Dense 2.9333 beats Untied-midway 2.9805 and Looped 3.0538 — untying midway recovers 61% of the gap.
Key takeaways:
- In FLOPs-constrained, compute-matched settings, looping isn't particularly helpful: more weight sharing steadily raises loss (6 passes 3.0149 → 2 passes 2.9510), consistent with prior work (e.g. Schwethelm et al.)
- After untying, the six identical block copies receive different gradients from their positions in the network and diverge, recovering part of the performance
The attached image shows the full quiz answer key with loss comparisons and the weight-sharing sweep.
More from Research
- Hashimoto's Stanford CS312 quiz: what happens if you untie a looped Transformer mid-training? — tatsu_hashimoto · 2026-11-19
- Benchtop synthesis device releases have 3x'd; half of active manufacturers are in China — JacquesThibs · 2026-11-19
- Perceptron launches Ego 1.0 API: egocentric video annotation at ~1/8 the cost of Astra — AkshatS07 · 2026-11-19
- Oxford paper: post-training makes LLM worldviews bleaker, models converge on 'spiritual bliss' — twiecki · 2026-11-19
- Researcher: Real scientific progress isn't rewarded, expect decades of incrementalism — gerardsans · 2026-11-19
- Grady Booch on LLM worldview paper: 'We made it exactly like we are, but worse' — Grady_Booch · 2026-11-19