By LeCun's math definition of extrapolation, everything a neural net does is extrapolation
burny_tech · x · 2026-09-02
The author challenges how to mathematically define "true extrapolation" and "invariant causal models" to back claims in a thread, noting that by Yann LeCun's mathematical definition, basically everything a neural network does is extrapolation.
The quoted tweet recalls that 3 years ago, after the Chinchilla paper (arXiv:2203.15556) sparked euphoria, several top physicists left CMTC for AI companies to 'understand' scaling behavior in LLMs. Chinchilla's key result: under a fixed compute budget, model size and training tokens should scale equally — its 70B/4x-data model significantly outperformed Gopher, GPT-3, and others.
More from AGI Musings
- AI now generates more math proofs than humans can verify; Tao spent days condensing a 90,000-line proof — every · 2026-09-02
- As formal reasoning gets dirt cheap, Fermi and Feynman tales look 'quaint' — akbirthko · 2026-09-02
- "We were drowning in slop long before AI": the case that AI slop is the hero — taherdhanera · 2026-09-02
- Felix Simon adds three more reasons AI persuasion won't dominate the real world — _FelixSimon_ · 2026-09-02
- AI is a scary-good persuader, but its real-world influence will be smaller than feared: Oxford researcher — _FelixSimon_ · 2026-09-02
- AI may replace workflows long before it replaces jobs — SuspiciousAir6358 · 2026-09-02