Language is a lossy compression: even the best models train on a thin residue of reality
yunta_tsai · x · 2026-09-27
The author argues that language isn't reality but a slow social compression of it: spoken English carries only tens of bits per second of new information—below a 56k modem—and text is only marginally denser, so nearly everything a nervous system registers never makes it into training corpora.
Tokens, in this framing, are an accident of human writing. Feed a model the uncompressed stream—electromagnetic waves and force rather than tokens—and the first problem shifts from prediction to deciding what counts as a unit. A mind perceiving the raw field would invent its own compression: events, invariants, causal structure, and geometries we don't yet have names for.
Related event: Language Is a Lossy Compression of Reality(2 posts)→
More from AGI Musings
- Malthus Saw It in 1798: Wealthy Avoiding Kids, Delayed Marriage Repeats Today — SydSteyerhart · 2026-11-16
- Runway CEO Predicts Fully AI-Generated Cultural Phenomenon by 2027 — c_valenzuelab · 2026-11-16
- '5% worse and 10% cheaper': the button with no vote is already everywhere — eigenrobot · 2026-11-16
- Effective Altruism is top-down and hierarchical, not a grassroots movement — ctjlewis · 2026-11-16
- Lawyers say clients' AI use creates unrealistic expectations and fake case law in settlements — hoofnagle · 2026-11-16
- Dev debunks 'rogue AI' hype: subsidized subscriptions can't fund expensive swarms — gerardsans · 2026-11-16