Surya Ganguli lecture traces the statistical-physics origins of scaling laws
burny_tech · x · 2026-10-18
A 1-hour Stanford lecture by Surya Ganguli, "The Origins of Scaling Laws": predicting LLM scaling exponents from measurable statistics of natural language, using statistical-physics ideas. Covers n-gram loss scaling collapse, the data threshold P(n)≈n^2β for looking back n words, power-law decay of word correlations, and why longer context ranges demand more training data.
More from Research
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18
- Brain implants are restoring speech to people with ALS, Science reports — EricMTrautmann · 2026-10-18
- Formal Verification of LLM-Generated Code: Slides from FATTW 2026 Talk — CatAstro_Piyush · 2026-10-18