Where the money goes per token: a diagram of LLM API cost anatomy
cneuralnetwork · x · 2026-11-01
A shared infographic breaks down the end-to-end cost of a single LLM API call: prompt → tokenise → prefill (once, builds the KV cache over all input tokens) → decode (repeated, generates output tokens using the KV cache).
You pay for three things: prefill compute on input tokens, decode compute on output tokens, and the memory to keep the KV cache — which scales with context length, explaining why long-context calls cost more.
More from Infra
- Before GPU Engineering: free 56-chapter open notebook on GPU architecture — techNmak · 2026-11-01
- FreeToken brings datacenter-scale MoE model serving to desktops, hits 15k stars — FlashML-org · 2026-11-01
- Musk claims xAI's capex efficiency is greatly underestimated — RachelVT42 · 2026-11-01
- AI bubble pops globally: China wipes out ¥23.5T, Nasdaq drops 3.46% in a day — Grady_Booch · 2026-11-01
- Strata engine underwhelms on 2× R9700: loses to llama.cpp despite higher t/s — Don_Reuter · 2026-11-01
- exo clusters your Macs into one AI machine to run frontier models locally — alex_verem · 2026-11-01