Zhipu's Tang Jie on Scaling Laws: Inference and MoE Shift Optima
jietang · x · 2026-08-19
Zhipu's Tang Jie provides a deep analysis of Scaling Law evolution. He notes that early Kaplan laws led to over-expansion of parameters, while Chinchilla corrected the compute-optimal ratio to 20:1. However, as inference costs dominate, the trend has shifted to smaller models trained longer (e.g., Llama-2).
MoE architecture further changed the target: total parameters govern knowledge retention, while activated parameters govern reasoning depth. Roberts (2025) found the optimal ratio is task-dependent, and pushing total parameters too high can actually degrade reasoning.
GLM-5.3 serves as a controlled experiment: keeping the base and parameters constant, it achieved non-marginal gains through one month of long-horizon RL and post-training, proving that beyond a parameter threshold, effective depth and post-training are key.
More from Models
- Qwen3.8 27B Breaks Size-Intelligence Curve, Resets Consumer Hardware Expectations — smtabatabaie · 2026-08-19
- Analysis Suggests OpenAI's 'Astra' Model Unaffected by RL Slowdown — haider1 · 2026-08-19
- Anthropic's protein design PR dissected: open-source models did most of the lifting — teortaxesTex · 2026-08-19
- Inkling-Small hits Agent Arena: matches Inkling performance at half the price — simonguozirui · 2026-08-19
- LFM2.5 Models and Embedding Now Available on OpenRouter — maximelabonne · 2026-08-19
- Will we run 30B+ parameter models fast on small GPUs in the future? — absurdother · 2026-08-19