LLMs see context as a landscape — your token choices shape agent behavior
_Stocko_ · x · 2026-11-20
The author offers an intuition: LLMs don't read context sequentially like a human — they 'see' it all at once, like a landscape, in which human tokens look distinct. The accompanying image shows how "Sure, ok. Go ahead." tokenizes into 4, 5, or 7 tokens with different IDs under GPT-4o's o200kbase tokenizer depending on casing and spelling. The takeaway: the care you put into your tokens affects your interaction with your agent more than you'd think.
More from coding & agent
- MCP Events extension finalized in beta, ready for client and server adoption — dsp_ · 2026-11-21
- Using Notion as an AI context layer with an external Claude skills-db — ivanhzhao · 2026-11-21
- Gaussian splats in Godot: full tutorial plus using Fable agent to plan a pixel apartment layout — XRarchitect · 2026-11-21
- Skip agent memory: a folder structure and index beats it — NewsLewis · 2026-11-21
- DevFest Pakistan 2026: deconstructing Gemini Deep Research with Antigravity 2.0 and Gemini 4 Neon — gerardsans · 2026-11-21
- mitsuhiko nudges GitHub to add an AI agent to 4-year-old merge=union feature request — mitsuhiko · 2026-11-21