Liam Fedus on Scaling Laws: FLOPs are Intelligence, Parameters are Knowledge
hsu_byron · x · 2026-08-20
Liam Fedus retweeted Jie Tang's history of scaling laws, adding context from the 2020 Switch Transformer experiments. They routed tokens to 1 out of 2048 experts, achieving 1.6T total parameters with <3B activated parameters. While this model achieved SOTA on TriviaQA with superior compute efficiency, it failed on reasoning tasks like SuperGLUE. Fedus concluded that the optimal token-per-parameter ratio is task-dependent, reinforcing Noam Shazeer's intuition: FLOPs are intelligence; parameters are knowledge.
More from Research
- 7 parameters that control every LLM response explained — blaizedsouza · 2026-08-20
- DeepLabCut Celebrates 9th Anniversary with 1.14M Installations — TrackingActions · 2026-08-20
- Opinion: Loss metric is a relic unfit for modern data scales — gerardsans · 2026-08-20
- Opinion: Broken scaling laws result from ignoring output fidelity — gerardsans · 2026-08-20
- Language understanding requires internal modeling of space-time, emerges earlier — RexDouglass · 2026-08-20
- Jaap Jumelet Wins 2025 EurAI Dissertation Award for LLM Structure Research — banazir · 2026-08-20