Zhipu's Tang Jie on Scaling Laws: Inference and MoE Shift Optima
jietang · x · 2026-08-19
Zhipu's Tang Jie provides a deep analysis of Scaling Law evolution. He notes that early Kaplan laws led to over-expansion of parameters, while Chinchilla corrected the compute-optimal ratio to 20:1. However, as inference costs dominate, the trend has shifted to smaller models trained longer (e.g., Llama-2).
MoE architecture further changed the target: total parameters govern knowledge retention, while activated parameters govern reasoning depth. Roberts (2025) found the optimal ratio is task-dependent, and pushing total parameters too high can actually degrade reasoning.
GLM-5.3 serves as a controlled experiment: keeping the base and parameters constant, it achieved non-marginal gains through one month of long-horizon RL and post-training, proving that beyond a parameter threshold, effective depth and post-training are key.
More from Models
- Meta open-sources Muse Spark Open agent model with day-0 TPU support in SGLang — BanghuaZ · 2026-10-16
- Saying "load-bearing" in your prompt makes Opus 5.5 stop holding back — zacharynado · 2026-10-16
- Americans get their Claude accounts banned while on business trips in China — dotey · 2026-10-16
- I priced every Grok Bot call: $20 to $200 plan in 5 days, long jobs eat 52% — jonathanmalkin · 2026-10-16
- NVIDIA releases Nemotron-3.5-Embed-1B, a multimodal embedding model covering 34 languages — jacek2023 · 2026-10-16
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16