TPOT of 20ms means 50 tokens/sec: a thread untangling LLM inference speed metrics
abhijithneil · x · 2026-09-04
A technical explainer thread breaks down core LLM inference performance metrics:
- TPOT (time per output token, sometimes written as ITL, inter-token latency): the gap between each token after the first — essentially your streaming speed.
- Per-user TPS = 1 / TPOT. With a TPOT of 20 milliseconds, a user sees 50 tokens per second.
- System TPS is the total tokens per second the server produces across everyone: 100 users each getting 50 adds up to a system throughput of 5000.
The thread's key point: distinguish single-user perceived speed from aggregate system throughput — the foundation of inference capacity planning.
Related event: Viral Thread Explains LLM Inference Metrics: TTFT, TPS and TPOT(4 posts)→
More from Infra
- AI-stock TL20 up 61% YTD as CoreWeave hits $104B backlog and Samsara jumps 14% — TiernanRayTech · 2026-09-04
- Nvidia A100 pre-training pipeline goes live on a 24/7 public livestream — wavefnx · 2026-09-04
- ADSP Episode 302: Mark Saroufim on PyTorch, GPU MODE, and automating AI research — blelbach · 2026-09-04
- TheDrummer returns with Artemis 31B v1/v1.1 and plans a crowdsourced inference platform — TheLocalDrummer · 2026-09-04
- Spotify's Portal Cut Claude Code Token Usage by 90% With a Two-Mode Router — rseroter · 2026-09-04
- Vyact: Open-Source Desktop Workspace Unifying Local LLMs, RAG, and Browser Context — vyact · 2026-09-04