Terence Tao: AI lacks its Shannon theory, so nobody knows scaling's limits
burny_tech · x · 2026-11-11
Terence Tao says we still cannot predict how many weights a model needs or how long training takes — it's all empirical rules of thumb because foundational concepts are missing. Communication has Shannon's information theory (bits, entropy), which is why we can stream megabytes per second and bound spectrum bandwidth, but there's no equivalent theory of intelligence or data complexity. So nobody knows AI's limits: scaling keeps working, but no one can say where it stops, because the mathematics doesn't exist yet. Tao says the field needs more mathematicians working on the theory of AI.
More from AGI Musings
- TissueMicroscopyBench: 49 evals show AI agents lack visual biology intuition under the microscope — anshulkundaje · 2026-11-12
- AI Circles Debate Whether a Lab Employee Strike Is the Fastest Way to Pause AI — jungofthewon · 2026-11-12
- Ethan Mollick: AI is the powerloom moment for intellectual craft work — emollick · 2026-11-12
- China's 100MW datacenters cost under $300M, one-fifth of US; Anthropic narrative shift predicted — max_paperclips · 2026-11-12
- The bitter, unreasonable effectiveness of reinforcement learning — burny_tech · 2026-11-12
- Reading list on novelty search: from the 1969 'Creative Porpoise' paper to Stanley & Lehman — dioscuri · 2026-11-12