Z.ai's GLM-5.3 Essay: Scaling Law Has More Dials Than Parameter Count
pmttyji · reddit · 2026-08-19
Z.ai published a long essay on scaling laws, framed around GLM-5.3 as a controlled experiment: same base, architecture, and total/activated parameters as GLM-5.2, with only one month of scaled long-horizon environments and RL — yielding non-marginal gains.
The essay traces the field's evolution: Kaplan et al. (2020) fit parameters growing 2.7:1 over data, driving GPT-3, Gopher, MT-NLG; Hoffmann et al. (2022) (Chinchilla) redid it across 400 models, finding 20 tokens per parameter optimal and co-growth at scale — making the trillion-parameter round a field-wide detour.
Later shifts: with inference dominating lifetime cost, optima moved to smaller, deliberately over-trained models (Llama-2-7B, Gemma-2-9B at 290 and 889 TPP); under MoE, total parameters govern knowledge while activated parameters and depth govern reasoning chains; Roberts et al. (2025) find optimal TPP is task-dependent — memorization favors parameters, reasoning favors data, and pushing total parameters at fixed TPP degrades reasoning.
Conclusion: beyond a threshold, capability comes from effective depth and post-training. Base size, pretraining data, and per-forward-pass compute remain open; next up may be mid-training or pre-training.
More from Models
- GPT-4o API dies January 14 as OpenAI refuses to release weights, pushing users to Qwen — mimi10v3 · 2026-10-16
- Gemini 4 Argon goes live in Antigravity with its own weekly quota for Pro subscribers — rseroter · 2026-10-16
- Opus 5.5 value may end the habit of using the top model for everything — adonis_singh · 2026-10-16
- Musk says Grok Bot is growing faster than anything he's seen, with no free tier — elonmusk · 2026-10-16
- Peter Yang: Opus 5.5 Still Struggles to Hold Its Ground When Pushed Back — petergyang · 2026-10-16
- Robot Startup Ditches VLAs for Frontier LLMs in a Customer Deployment — ihorbeaver · 2026-10-16