Engineers Using LLMs Should Understand Tokens, Context Windows and Hallucinations
_jaydeepkarale · x · 2026-11-12
A narrative primer arguing that engineers who use LLMs daily should understand what happens under the hood—not to train models, but to debug behavior, control costs, and make better building decisions.
- Tokenization: models read tokens, not words, so a 1,000-word prompt isn't 1,000 tokens—hence token-based API pricing.
- Context windows: limited capacity means earlier info (e.g., a database schema from 50 messages ago) gets dropped or summarized, causing "forgetting."
- Embeddings: semantic vector representations enable meaning-based retrieval, e.g., matching "reduce API latency" to docs about "improving response times."
- Hallucinations: LLMs predict likely next tokens without fact-checking, so plausible answers can be fabricated—nonexistent functions, fake citations.
- Temperature: probabilistic generation explains inconsistent outputs; lower temperature yields steadier results, higher allows variation.
The thread (truncated at how models pick among tokens) frames each concept as a story that explains a real limitation engineers encounter.
More from coding & agent
- Matt Pocock's Home Repo pattern: run coding agents across multiple repos, setup prompt included — mattpocockuk · 2026-11-12
- Nearly one-shot: Fable 5.2 with Opus subagents creates Taj Mahal explainer video — CurieuxExplorer · 2026-11-12
- Princeton & Anthropic find the Memory Trap: agents skip verification on a $1.2M payment — aigclink · 2026-11-12
- DataTalks.Club launches free 5-week course on production AI agents for generative media — Al_Grigor · 2026-11-12
- Claude designs 3 working gadgets from one-paragraph prompts, specs to firmware in a day — JafarNajafov · 2026-11-12
- Dev uses hourly "Ok" scheduled task to keep Claude's 5-hour usage window always fresh — umeshonai · 2026-11-12