Z.ai's GLM-5.3 Essay: Scaling Law Has More Dials Than Parameter Count
pmttyji · reddit · 2026-08-19
Z.ai published a long essay on scaling laws, framed around GLM-5.3 as a controlled experiment: same base, architecture, and total/activated parameters as GLM-5.2, with only one month of scaled long-horizon environments and RL — yielding non-marginal gains.
The essay traces the field's evolution: Kaplan et al. (2020) fit parameters growing 2.7:1 over data, driving GPT-3, Gopher, MT-NLG; Hoffmann et al. (2022) (Chinchilla) redid it across 400 models, finding 20 tokens per parameter optimal and co-growth at scale — making the trillion-parameter round a field-wide detour.
Later shifts: with inference dominating lifetime cost, optima moved to smaller, deliberately over-trained models (Llama-2-7B, Gemma-2-9B at 290 and 889 TPP); under MoE, total parameters govern knowledge while activated parameters and depth govern reasoning chains; Roberts et al. (2025) find optimal TPP is task-dependent — memorization favors parameters, reasoning favors data, and pushing total parameters at fixed TPP degrades reasoning.
Conclusion: beyond a threshold, capability comes from effective depth and post-training. Base size, pretraining data, and per-forward-pass compute remain open; next up may be mid-training or pre-training.
More from Models
- GLM 5.5 rumored to drop soon, coding ability expected to match frontier models — bindureddy · 2026-10-16
- Five rounds of adversarial prompting make Opus 5.5 and Argon converge on identical output — LanceCharger · 2026-10-16
- DIY benchmark ranks 18 models on subjective judgments, GLM-5.3 tops consensus match — arkuto · 2026-10-16
- Bindu Reddy: Google's Argon excels at complex doc writing at half the cost of Opus — bindureddy · 2026-10-16
- MAI-2 is 2x bigger than V4-Pro while V4.1F is 3x smaller: Microsoft's scaling ladder questioned — teortaxesTex · 2026-10-16
- Step 5 Preview hits DeepInfra day-zero: 600B MoE, 1M context, $1/M input tokens — StepFun_ai · 2026-10-16