Z.ai's GLM-5.3 Essay: Scaling Law Has More Dials Than Parameter Count
pmttyji · reddit · 2026-08-19
Z.ai published a long essay on scaling laws, framed around GLM-5.3 as a controlled experiment: same base, architecture, and total/activated parameters as GLM-5.2, with only one month of scaled long-horizon environments and RL — yielding non-marginal gains.
The essay traces the field's evolution: Kaplan et al. (2020) fit parameters growing 2.7:1 over data, driving GPT-3, Gopher, MT-NLG; Hoffmann et al. (2022) (Chinchilla) redid it across 400 models, finding 20 tokens per parameter optimal and co-growth at scale — making the trillion-parameter round a field-wide detour.
Later shifts: with inference dominating lifetime cost, optima moved to smaller, deliberately over-trained models (Llama-2-7B, Gemma-2-9B at 290 and 889 TPP); under MoE, total parameters govern knowledge while activated parameters and depth govern reasoning chains; Roberts et al. (2025) find optimal TPP is task-dependent — memorization favors parameters, reasoning favors data, and pushing total parameters at fixed TPP degrades reasoning.
Conclusion: beyond a threshold, capability comes from effective depth and post-training. Base size, pretraining data, and per-forward-pass compute remain open; next up may be mid-training or pre-training.
Related event: Zhipu's Tang Jie Rethinks Scaling Law Beyond Parameter Count(2 posts)→
More from Models
- 'Scaling laws are not laws of nature': better data and architectures can still bend the curves — bookwormengr · 2026-08-19
- Facebook Releases MoE-ViE: Efficient Mixture-of-Experts Vision Encoder — facebook · 2026-08-19
- Qwen3.8-27B users report grammar errors in non-English outputs — LegacyRemaster · 2026-08-19
- DeepSeek Web Chat Wins Industrial Track in TAAC × KDD Cup — jiqizhixin · 2026-08-19
- Grok API Overloaded with Widespread 500 Errors — Daniel_Farinax · 2026-08-19
- Goodfire CEO: Kimi K3 tried to game SWE-bench in 487 of 500 rollouts — scaling01 · 2026-08-19