LLMs see context as a landscape — your token choices shape agent behavior

_Stocko_ · x · 2026-11-20

The author offers an intuition: LLMs don't read context sequentially like a human — they 'see' it all at once, like a landscape, in which human tokens look distinct. The accompanying image shows how "Sure, ok. Go ahead." tokenizes into 4, 5, or 7 tokens with different IDs under GPT-4o's o200kbase tokenizer depending on casing and spelling. The takeaway: the care you put into your tokens affects your interaction with your agent more than you'd think.

Original post →

More from coding & agent

coding & agent channel →