Jie Tang on scaling history: FLOPs were intelligence, parameters were knowledge
cedric_chee · x · 2026-08-21
Jie Tang highlighted a discussion on the history of scaling laws. Liam Fedus recalled 2020's Switch Transformers: fewer than 3B activated parameters out of 1.6T total (comparable to today's frontier models). It beat T5 on C4 perplexity with far less compute and set a TriviaQA SOTA, yet failed badly on reasoning tasks like SuperGLUE.
The lesson: the optimal tokens-per-parameter ratio is highly task-dependent. Or as Shazeer intuited: FLOPs were intelligence; parameters were knowledge.
More from Models
- API Model Gains Vision Capabilities in Major Update — teortaxesTex · 2026-08-21
- Experts question Anthropic's trust in Claude's simulation excuses — GarrisonLovely · 2026-08-21
- LLM German Output Cringed: Reads Like It Was Written by Olaf Scholz — DominiqueCAPaul · 2026-08-21
- DeepSeek V4 Flash adds vision capabilities, now live — TheCryptoCat75 · 2026-08-21
- DeepSeek V4-vision-exp launches API with ultra-fast speed and low cost — teortaxesTex · 2026-08-21
- Fastest NVFP4 quant of Qwen3.8 27B released, 50% faster than Q4 on compatible hardware — ionsago · 2026-08-21