Liam Fedus on Scaling Laws: FLOPs are Intelligence, Parameters are Knowledge
hsu_byron · x · 2026-08-20
Liam Fedus retweeted Jie Tang's history of scaling laws, adding context from the 2020 Switch Transformer experiments. They routed tokens to 1 out of 2048 experts, achieving 1.6T total parameters with <3B activated parameters. While this model achieved SOTA on TriviaQA with superior compute efficiency, it failed on reasoning tasks like SuperGLUE. Fedus concluded that the optimal token-per-parameter ratio is task-dependent, reinforcing Noam Shazeer's intuition: FLOPs are intelligence; parameters are knowledge.
More from Research
- Transformer trained on flying bat recordings yields hippocampus 'digital twin' revealing hidden structure — patrickmineault · 2026-10-16
- One year of Recursive Language Models: GPT-5-mini beats GPT-5 by 110% on 132k-token tasks, cheaper too — a1zhang · 2026-10-16
- MIT PhD Sharut Gupta joins DeepMind; Isola hails her representational alignment defense talk — phillip_isola · 2026-10-16
- 36 primitives, just 10 parameters each: a remarkably precise T-pose body fit — letandrewcook · 2026-10-16
- Interp researcher pushes back: if you can't simulate a neuron, you haven't explained it — nabla_theta · 2026-10-16
- MLP-Splatting (ECCV 2026) code released: MLPs as light-field primitives instead of Gaussians — AjdDavison · 2026-10-16