Sauers_ rebuts "LLMs are inherently discrete": tokens are readouts of a continuous process

Sauers systematically rebuts the popular claim that "LLMs output discrete tokens, so they are fundamentally discrete and discontinuous, unlike humans' smooth, continuous stream of thought." His core argument: discreteness is a surface property of the output interface, not of the system itself.

Confirmed

• Tokens are merely intermediate readouts of the underlying continuous computation—snapshots of a continuously evolving internal state taken at sampling moments, not the process itself.
• He backs this with an information-theoretic calculation: with a 400k context, the identity information of past tokens totals roughly 6,885,242 bits (400,000 × log₂ of a 151k vocabulary).
• Token identity information turns out to be a tiny fraction of available information: on the order of 1% to 0.001% (0.0008%) depending on framing—the vast majority of information lives in the continuous internal state, not the token sequence.

Why it matters

• "LLMs are discrete, so they can never resemble human thinking" is often used as a qualitative judgment on LLM capability ceilings; if tokens are just low-bandwidth readouts of a continuous process, that argument collapses, and the discussion should shift to the properties of the internal continuous dynamics.
• The information ledger offers a testable, quantitative lens: assessing an LLM's expressive power should not hinge on the token stream, but on the information carried by its continuous internal states.

Note: All five posts in this cluster come from the same author, Sauers, posted on 11-11 as variations of the same argument; the above merges them into a single statement.

2026-11-11 ~ 2026-11-11 · 5 related posts

• LLMs aren't discrete: tokens are just ~0.00001% of the continuous process — Sauers_ · 2026-11-11
• Tokens are just readouts: why LLMs aren't fundamentally discrete systems — Sauers_ · 2026-11-11
• Tokens are only ~1% of the story: LLMs compute continuously, discrete output is readout — Sauers_ · 2026-11-11
• [source] LLMs Aren't Fundamentally Discrete: Tokens Are Just Readouts of a Continuous Process — Sauers_ · 2026-11-11
• [source] Tokens Are Just 0.0008% of a 400k Context: The KV Cache Math — Sauers_ · 2026-11-11