CS312 course post explains sharpness and the edge of stability in LLM pretraining
tatsu_hashimoto · x · 2026-10-31
This week's Stanford CS 312 lesson tackles one of the most unintuitive concepts in pretraining: sharpness.
- The article explains what sharpness is and how it shows up in real training runs—gradient descent should theoretically diverge, yet loss hovers near the edge of stability
- It also outlines the field's open questions about sharpness
- A solid primer for anyone studying pretraining dynamics and optimization
More from AGI Musings
- Engineer rates 10 kinds of work on 4 automation levels: only PM stays human — kieranklaassen · 2026-10-31
- 1946 sci-fi 'A Logic Named Joe' predicted uncensored LLMs and jailbreaking — EffectiveMedium2683 · 2026-10-31
- ML veteran Burkov: superintelligence abundance would drain Earth's remaining resources — burkov · 2026-10-31
- Santa Fe Institute's Eric Smith dismantles anthropocentrism in careful terms — danfaggella · 2026-10-31
- Bindu Reddy predicts 12 months of AI trends: tokens -80%, usage 10x, coding jobs shrink — bindureddy · 2026-10-31
- Accumulating wealth may help build a civilization where wealth itself becomes meaningless — Dr_Singularity · 2026-10-31