Co-occurrence is the mathematical backbone of token-driven LLMs
gerardsans · x · 2026-10-04
Arguing that co-occurrence isn't the full picture but an essential ingredient, the author explains that weights and biases are compressed distributions from gradient descent over a corpus, then breaks down the full chain: embedding creation, inference dynamics (tokenisation, positional encoding, pre-fill, attention/MLP layers, sampling) and the autoregressive loop.
Related event: Debate: Co-occurrence Statistics as the Mathematical Foundation of LLMs(3 posts)→
More from Research
- Paper Finds Personal Agents Get Worse as Memory Notes Pile Up — rohanpaul_ai · 2026-10-05
- Micro entities and individual inventors are far likelier to use AI in patent applications, new paper finds — robseamans · 2026-10-05
- Elephant rumble choruses encode social context through acoustic convergence, study finds — abenitezburraco · 2026-10-05
- Swarm added just +0.05: splitting context into blocks did the real work for tiny local models — deadatreides1 · 2026-10-05
- Sinclair's Harvard lab publicly demos resetting colon cancer cells via 3 reprogramming genes — tomchapin · 2026-10-04
- Diverse Approach Sampling Beats IID Self-Training on Hard Problems — PMinervini · 2026-10-04