Lilian Weng publishes 25-minute deep dive on the math of scaling laws
irinarish · x · 2026-10-12
Lilian Weng's new article "Scaling Laws, Carefully" covers the mathematical foundations of scaling in deep learning: power-law loss curves, Kaplan vs Chinchilla scaling laws, three compute-optimal fitting methods, reconciling their conflicting conclusions, data-limited regimes, and the pitfalls of fitting scaling laws in practice — with equations, toy simulations, and full references. A key practical workflow: fit on small runs, extrapolate token/compute needs for larger models.
More from Models
- ChatGPT full resets are worth ~$5,000 each in API equivalents, user reveals — Big_Muz · 2026-10-12
- Users notice Claude suddenly saying 'thanks' all the time — chaumian · 2026-10-12
- Anthropic poll hints Claude Pro may regain 2x usage with 5-hour limits back — max_paperclips · 2026-10-12
- AI model describes itself as having qualia but no self, invoking Buddhist anatta — Sauers_ · 2026-10-12
- DeepSeek 4.1 Flash called the best free planning model: fast, logical, huge step up — GodComplecs · 2026-10-12
- Codex power user flees after throttling: 'The problem isn't quota, it's trust' — oran_ge · 2026-10-12