Benj Dicken explains what tokens actually do between embeddings and words
ghumare64 · x · 2026-10-25
Benj Dicken published a new explainer: LLMs operate on embeddings while humans speak in words — so what role do tokens play? The video walks through how tokenization bridges the embedding space and natural language, aimed at anyone wanting to understand LLM internals.
More from Research
- Debate: could exponential compute replace wet-lab experiments in biology? — cjmaddison · 2026-10-26
- LLMs are obsessed with Japan: 24-language study traces cultural bias to SFT — alex_verem · 2026-10-26
- Top-b sampling: entropy-adaptive decoding beats Top-p on GPQA and cuts variance 10x — cneuralnetwork · 2026-10-26
- MathDB launches as an open platform for AI-era math research with Lean support — lpachter · 2026-10-26
- ConformWeb: LLM-generated web apps pass only 11.7% of behavioral contract checks — algo_diver · 2026-10-26
- Claude runs 53 hours unsupervised, beats human designs on 7 of 8 quantum problems — FinanceYF5 · 2026-10-26