Your LLM never sees your prompt — it sees tokens: a tokenization primer
mdancho84 · x · 2026-10-31
Author mdancho84 argues data scientists moving into AI engineering skip a fundamental topic: the model sees tokens, not your prompt.
Key points:
- Learn tokenization before agents, RAG, MCP, or fine-tuning
- It quietly affects context window limits, API cost, code generation, numbers/identifiers, multilingual performance, embeddings, chat templates, and training
- Common misconceptions: BPE is not automatically byte-level; SentencePiece is not a tokenization algorithm; token IDs carry no semantics
- Looks like an implementation detail until you're debugging and realize the model received information differently than you assumed
The attached image is a handbook outline covering UTF-8, BPE vs byte-level BPE, SentencePiece/Unigram/WordPiece, special tokens and chat templates, multilingual/code tokenization, and context length/cost coupling.
More from coding & agent
- 174-line mental schema + Claude Code: an agent workflow that builds entire slide decks — tianshi_li · 2026-11-01
- Immunologist uses OpenAI Codex to simulate single-cell drug experiments — DeryaTR_ · 2026-11-01
- Stripe Link goes live in Agent Picnic to manage AI agent connections and permissions — jeff_weinstein · 2026-11-01
- BossBot books pickleball courts at midnight daily, beating the reservation rush — chrisfleck · 2026-11-01
- OpenAI's Astra 5.5 runs massive code patches just to list Devcon side events — devanshmehta · 2026-11-01
- Curated skills collection lets you run agents on your own harness and subscriptions — jasonkneen · 2026-11-01