Streaming LLM metrics explained: TPOT is inter-token latency, the real streaming speed
abhijithneil · x · 2026-09-04
In an ongoing explainer thread, the author breaks down the core latency metrics for LLM serving:
- TTFT (time to first token): how long from sending the request to seeing the first word, mostly prefill time plus queueing delay.
- TPOT (time per output token): the gap between each token after the first — effectively your streaming speed, also written as ITL (inter-token latency); they are the same thing.
- TPS: tokens per second.
These metrics form the baseline vocabulary for evaluating inference service responsiveness.
Related event: Viral Thread Explains LLM Inference Metrics: TTFT, TPS and TPOT(4 posts)→
More from Infra
- AI-stock TL20 up 61% YTD as CoreWeave hits $104B backlog and Samsara jumps 14% — TiernanRayTech · 2026-09-04
- Nvidia A100 pre-training pipeline goes live on a 24/7 public livestream — wavefnx · 2026-09-04
- ADSP Episode 302: Mark Saroufim on PyTorch, GPU MODE, and automating AI research — blelbach · 2026-09-04
- TheDrummer returns with Artemis 31B v1/v1.1 and plans a crowdsourced inference platform — TheLocalDrummer · 2026-09-04
- Spotify's Portal Cut Claude Code Token Usage by 90% With a Two-Mode Router — rseroter · 2026-09-04
- Vyact: Open-Source Desktop Workspace Unifying Local LLMs, RAG, and Browser Context — vyact · 2026-09-04