An LLM is learning to fill a giant next-token matrix, stored sparsely as training data
iamtrask · x · 2026-11-12
Andrew Trask offers a clean mental model: imagine a matrix whose rows are every possible token combination in a million-token context and whose columns are next-token frequency distributions — that's what an LLM learns to fill. We never fully render it because that wastes space; it's stored in sparse form, which is the token training data itself.
Related event: iamtrask Explains LLMs as a Giant Prediction Matrix(2 posts)→
More from Research
- Why do LLMs talk as if they have bodies? Mechinterp study targets phantom embodiment — repligate · 2026-11-12
- Scaling H2R workshop at CoRL 2026 probes scaling laws for human-to-robot learning — mangahomanga · 2026-11-12
- 3 GitHub repos worth reading for the code: nanoGPT, tinygrad, llama.cpp — cneuralnetwork · 2026-11-12
- 38,101 ICLR 2027 Submissions Scraped: 2,209 Papers Share Just 750 Acronym Names — IanArawjo · 2026-11-12
- LLMs sit by default: 91-98% of Qwen3 models describe their imagined body as seated — repligate · 2026-11-12
- Sparse Modeling textbook by Irina Rish resurfaces as a classic intro to sparsity theory — FrnkNlsn · 2026-11-12