Learning is compression: a long-form essay on manifold hypothesis and why intelligence is possible at all
burny_tech · x · 2026-11-09
A thoughtful long-form essay arguing why machine learning works at all:
- Manifold hypothesis: real-world data nominally lives in millions of dimensions, but meaningful data concentrates in tiny low-dimensional structured regions, constrained by identity, pose, lighting, physics, etc.
- Precondition for intelligence: the world must have predictable structure; in a white-noise universe no brain or Transformer could learn anything from the past.
- Learning is compression: models don't store a billion copies of faces but learn generative structure over latent variables — closely aligned with Kolmogorov complexity intuition.
- Uneven distributions aren't enough: what matters is exploitable dependencies between variables, measured by mutual information.
A coherent synthesis of manifold hypothesis, information theory, and compression as the foundation of intelligence.
More from AGI Musings
- METR says it has 'convincing evidence' AI speeds up research — but the details are redacted — peterwildeford · 2026-11-09
- OpenAI co-founder Greg Brockman: frontier models with test-time compute are becoming the engine of scientific discovery — gdb · 2026-11-09
- Long-term memory vs. search is like L1 cache vs. RAM, argues zetalyrae — zetalyrae · 2026-11-09
- AI is eroding traditional moats: years of advantage now replicated in days — DeryaTR_ · 2026-11-09
- Epoch AI: 2027 chip supply could run 30–170M frontier AI agents, or 1.9B cheaper ones — Dr_Singularity · 2026-11-09
- White paper: moving beyond the AI compute arms race as big tech faces historic spending losses — ReportProfessional57 · 2026-11-09