Stanford's Potts points to Figure 8 of "Why Larger Models Learn More": 1B models overwrite old copies
ChrisGPotts · x · 2026-10-24
Stanford professor Chris Potts confirms an empirical observation about model scale and knowledge acquisition timing, pointing to Figure 8 of the paper "Why Larger Models Learn More," with a visualization made by Ekdeep Ambekar.
- Core observation: at 500B training tokens, the 1B model sits lowest on every row — small models cover knowledge copies in the training corpus much earlier.
- Potts endorses the commenter's hypothesis that a small model overwrites more once the previous copy is far back in context, explaining scale differences in knowledge memorization timing.
- Mechanistic empirical work on why larger models learn more.
More from Research
- Action learning: score whole trajectories with no forces or simulator, 2000× lower error — LeoTZ03 · 2026-10-24
- BeNice! method shares KV-cache across loops, scaling looped LLM depth without extra memory — stochasticchasm · 2026-10-24
- 215-state Turing machine proven to halt iff ZFC is inconsistent, verified in Metamath Zero — teortaxesTex · 2026-10-24
- OmniRay from JHU hits SOTA camera-agnostic monocular 3D reconstruction from one image — kwangmoo_yi · 2026-10-24
- AWS multi-turn RL guide: $339 lifts gpt-oss-20b Wordle solve rate 61%→84% — AWS ML Blog · 2026-10-24
- Eric Horvitz spotlights Stanford's tool-making research philosophy behind optogenetics — erichorvitz · 2026-10-24