The subtle KV cache detail most people get wrong: tokens generate before entering the cache
techNmak · x · 2026-09-18
A clear explanation of a common KV cache misconception in decoder-only Transformers: not every token flows through the LLM the same way.
- Prefill: the prompt is processed once, each attention layer stores K/V for all prompt positions, and the final position's representation yields logits for y1.
- The subtle part: selecting y1 doesn't put it in the cache. On the next step, y1 enters as position 5; its Q attends to cached K/V from positions 1–4 plus its own current K/V, which then extends the cache.
- The process repeats: past K/V are reused, the cache grows as new positions are processed.
Key takeaway: a token can be generated before it has been processed into the KV cache — a distinction that matters a lot for understanding decoding.
More from Research
- CARLA driving expert with privileged maps trains vision-based imitation policy — abursuc · 2026-09-18
- AMP reframes robot manipulation as pixel classification to dodge action-space explosion — jiqizhixin · 2026-09-18
- PWNN Paper: Neural Networks as Remote Fault Injectors to Steal AES Keys From Cloud FPGAs — chaumian · 2026-09-18
- Camera-Only Autonomous Drive Through the Pyrenees: Lessons From a Small PhD Team — abursuc · 2026-09-18
- ACE-Data-0 hits 120K downloads in first month: 17M-frame multimodal embodied AI dataset — liuziwei7 · 2026-09-18
- Fei-Fei Li: 540 million years of vision evolution drove the development of intelligence — rohanpaul_ai · 2026-09-18