Why larger models learn more: capacity competition makes small models forget rare tasks
ChrisGPotts · x · 2026-10-23
Potts explains that Hardness proxies whether an example is in-distribution when encountered (more in-distribution → more memorization; Huang et al. 2024 §4.4), but it had comparatively low predictive value in their experiments. A unifying theory comes from Huang et al. 2026, Why Larger Models Learn More: limited capacity causes gradient interference, making models gradually forget infrequent, unusual tasks while retaining frequent ones — explaining both memorization profiles and Hardness modulation. Validated by pretraining OLMo models (4M–4B params): small models allocate neurons to high-frequency/low-complexity tasks and fail on rare ones even when expressible solutions exist; larger models reduce interference, letting rare-task features accumulate.
Related event: Stanford Team Shows Three Variables Can Predict LLM Verbatim Memorization(6 posts)→
More from Research
- Ropedia A0: code-only robot policy hits 70.24% on RoboDojo, 75% first-try sim2real — liuziwei7 · 2026-10-24
- Holi-World turns egocentric video into 3D/4D data: 4B model beats zero-shot 27B — liuziwei7 · 2026-10-24
- DecisionFormer open-sourced: fine-tune calibrated decision models on any LLM backbone — kalyan_kpl · 2026-10-24
- RLCR trains LLM agents as organizations: emergent hierarchies and division of labour — algo_diver · 2026-10-24
- Simplified Matching Distillation: few-step diffusion without adversarial training, validated up to 33B — ArashVahdat · 2026-10-24
- tszzl predicts stunning biology breakthroughs within years, cancer included — tszzl · 2026-10-24