Zhipu's Tang Jie on Scaling Laws: Inference and MoE Shift Optima
jietang · x · 2026-08-19
Zhipu's Tang Jie provides a deep analysis of Scaling Law evolution. He notes that early Kaplan laws led to over-expansion of parameters, while Chinchilla corrected the compute-optimal ratio to 20:1. However, as inference costs dominate, the trend has shifted to smaller models trained longer (e.g., Llama-2).
MoE architecture further changed the target: total parameters govern knowledge retention, while activated parameters govern reasoning depth. Roberts (2025) found the optimal ratio is task-dependent, and pushing total parameters too high can actually degrade reasoning.
GLM-5.3 serves as a controlled experiment: keeping the base and parameters constant, it achieved non-marginal gains through one month of long-horizon RL and post-training, proving that beyond a parameter threshold, effective depth and post-training are key.
More from Models
- Grok Bot Ran a 12-Hour Data Analysis Nonstop, Handling API Rate Limits Itself — Daniel_Farinax · 2026-10-16
- One year of Recursive Language Models: GPT-5-mini beats GPT-5 by 110% on 132k-token tasks, cheaper too — a1zhang · 2026-10-16
- Users Report Claude Models Suddenly More Compliant After Forced Compaction — RealSharpNinja · 2026-10-16
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Users Say Krea 2 Output Degraded Sharply Since Krea 3 Beta Launched — szywis · 2026-10-16
- LLM-as-a-Verifier lifts Nemotron 3.5 Super by 9-10 points on agentic benchmarks — simonguozirui · 2026-10-16