Tang Jie: Trillion Params Was a Detour, Scaling Laws Shift
青稞AI · wechat · 2026-08-20
Tsinghua Prof. Tang Jie analyzes the evolution of scaling laws:
- Ratio Correction: Kaplan's 2.7:1 param/data ratio misled the industry toward trillion-parameter models; Chinchilla proved the compute-optimal ratio is 20 tokens/param, with both growing equally.
- Inference Cost Dominates: With frequent inference, the optimal solution shifts to smaller, over-trained models.
- MoE Separation: Total params determine memory; activated params/effective depth determine long-horizon reasoning. Stacking total params can hurt reasoning.
- Post-training Gains: GLM-5.3 experiments show significant gains from one month of long-horizon RL post-training, confirming that scaling has multiple knobs.
More from Models
- Meta's coding-optimized open-weight Muse Spark Open lands on Modal with 1M context — AAAzzam · 2026-10-16
- Google launches Gemini 4 Argon frontier model on Cloud with 10 official demos — Saboo_Shubham_ · 2026-10-16
- RL practitioner: ~10,000 tasks still stump every frontier coding model — max_paperclips · 2026-10-16
- SpecID eval falls: Gemini 3.5 Flash scores 67%, beating its human creator at Zimbabwean species ID — johnowhitaker · 2026-10-16
- Researcher criticizes Claude for lecturing users and refusing critical takes on tech firms — BlancheMinerva · 2026-10-16
- Chart: Meta went 492 days without an open-weight release before Muse Spark Open — demian_ai · 2026-10-16