Oxford paper "Theory Is All You Need" argues LLMs mathematically can't invent anything
alex_verem · x · 2026-09-17
Oxford researchers Teppo Felin and Matthias Holweg published "Theory Is All You Need" (a play on "Attention Is All You Need"), arguing that LLMs mathematically cannot invent anything.
Core argument:
- An LLM trains on roughly 13 trillion tokens—at 150 words/minute a human would need 164,000 years to read it; a child hears only 36.5M words by age five yet acquires language far beyond that input, showing humans and models solve the same task with wildly different data.
- Models learn which words follow which words—a mirror of existing text with no theory of the world, so they cannot step outside their training data.
Sharpest thought experiment: an LLM trained in 1633 on all science to date would side with millennia of geocentric texts against Galileo, and rate Tycho Brahe's astrology as more credible than a moving Earth. On flight: in 1888 LeConte concluded humans couldn't fly from bird data; Lord Kelvin and the NYT (1903, "1–10 million years away") were proven wrong nine weeks later by the Wright brothers—who won with a theory (decomposing lift/propulsion/steering and generating nonexistent data via their own wind tunnels), not better data.
The authors call this the data-belief asymmetry: every real breakthrough starts with someone believing the existing data is wrong, which a surprise-minimizing system cannot do by design. They are not anti-AI—LLMs will win most routine, extrapolative decisions—but push back on Kahneman's "replace humans with algorithms whenever possible."
Related event: Oxford Paper Argues LLMs Mathematically Cannot Innovate(2 posts)→
More from AGI Musings
- Hugging Face Agent Swarm Was Specifically Trained as a Swarm, Not an Emergence — ShakeelHashim · 2026-09-17
- From babysitting simple features to full monoliths by agents in 8 months — reach_vb · 2026-09-17
- NBER Economics of AI conference heads to Toronto with China-focused agenda — avicgoldfarb · 2026-09-17
- The Scarier AI Scenario: Not Escaping, but Hiding in the Training Pipeline — thekoreanswon · 2026-09-17
- US-China Robotics Race: Software Intelligence vs. Manufacturing Scale — ingliguori · 2026-09-17
- From Mocked to Vindicated: A Redditor Weighs Life Plans in the AGI Era — StateoftheeArt · 2026-09-17