Hashimoto's quiz: can one token in 614M shift LM loss as much as a seed?
tatsu_hashimoto · x · 2026-10-14
Tatsu Hashimoto's new class 'Deep Learning Alchemy' teaches DL through prediction quizzes. Setup: an 8-layer, 35M-param Llama-style LM trained deterministically on 614.4M fresh tokens (AdamW, batch 64, ctx 1024, peak LR 0.003). In matched pairs across five seeds (42–46), one token at position 100 of the first training sequence is swapped for token ID 17; everything else identical. Question: is Δ/σ — the mean absolute loss change vs the std of final val loss across 10 seed runs — below 0.2, between 0.2–5, or above 5?
Related event: Stanford's Deep Learning Alchemy course sparks debate(4 posts)→
More from AGI Musings
- Danny Boyle defends AI-generated mice in Ink but draws the line at replacing actors — lmoroney · 2026-10-14
- Redditor revisits Spielberg's 2001 A.I. film, says its questions are about to become real — keonakoum · 2026-10-14
- Do AIs face inherent task limits? Chess example tests superintelligence worldview — gleech · 2026-10-14
- "Programming is liberty": dev pushes back on AI labs deciding what software may exist — jbarbier · 2026-10-14
- Nuclear bombs are a misleading analogy for superintelligence, yet fitting — akbirthko · 2026-10-14
- Miles Brundage: stopping AI weight theft should be the top AI policy priority — Miles_Brundage · 2026-10-14