Ex-OpenAI Researcher: Original Scaling Law Paper Had Fatal Flaw
新智元 · wechat · 2026-07-05
Former OpenAI researcher Diogo Almeida points out a fundamental bug in OpenAI's 2020 Scaling Law paper: it used a fixed number of tokens and cosine learning rate decay for models of all parameter sizes. This left large models severely "undernourished" while smaller ones were "overfed," leading to the erroneous conclusion that "prioritizing parameter scaling" was best. DeepMind researcher Sander Dieleman reshared the post, noting that this bug likely caused the industry to waste massive amounts of compute power on "oversized, under-trained" models.
DeepMind's 2022 Chinchilla paper overturned this conclusion, arguing that model parameters and data volume should be scaled equally, at a ratio of roughly 20 tokens per parameter. However, subsequent research revealed that the Chinchilla paper itself had a bug—the Huber loss in the optimizer was averaged per sample rather than summed, causing the fitting to terminate early. The paper correcting the bug came with another bug.
The article points out that Scaling Laws are essentially empirical fitting curves, not ironclad laws of physics. This history exposes a deeper issue: the entire AI industry once treated a flawed curve as a "first principle," basing massive compute allocation decisions on it.
Related event: Ex-OpenAI Researcher Finds Fatal Bug in Original Scaling Laws Paper(2 posts)→
More from Infra
- Nvidia gets mocked as “the leading open-source AI company” while repo chart shows it ahead — AccBalanced · 2026-07-27
- $8 ESP32-S3 runs a 28.9M-parameter LLM fully offline at 9.5 tokens per second — yangyi · 2026-07-27
- YC talk on BCI x AI says infrastructure is what really determines speed — garrytan · 2026-07-27
- A 13B model ran on a no-GPU PC by paging weights from SSD via llama.cpp — ID_R_McGregor · 2026-07-27
- llama.cpp warns that GGUFs made before a recent change must be regenerated — EconomySerious · 2026-07-27
- RTX 5090 local tests show Qwen Q6 can drop to 15 tok/s at 80k context — LFAdvice7984 · 2026-07-27