How to Estimate tokens/sec on Your Hardware: The VRAM Bandwidth Formula
Pyrolistical · reddit · 2026-09-05
A Reddit user shares a practical method for estimating local LLM decoding speed. Key insight: the decoding bottleneck is memory bandwidth, not compute — every generated token requires reading all model weights plus the KV cache from VRAM.
Basic formula (dense models): TG/s = VRAM GB/s ÷ model weights GB
Worked example: Qwen3.8 27B Q4KXL weighs 16.8 GB (excluding MTP layer and input embedding table); on an AMD Radeon AI PRO R9700 (637 GB/s), the theoretical max is 637/16.8 ≈ 38 TG/s. The author measures 29 TG/s with llama.cpp — 76% of ideal.
Full formula with KV cache: TG/s = VRAM GB/s ÷ (weights GB + KV cache GB/token × context length). Qwen3.8 27B's BF16 KV cache costs 64 KB per token. Rearranged, this yields a curve of TG/s-per-GB/s vs context size. For a 5090 (1.8 TB/s): 106 TG/s max at short context, dropping to 53 TG/s at 256k context.
Caveats: assumes everything fits in VRAM; simplified formula is dense-only; speculative decoding stacks on top; these are theoretical ceilings — real-world numbers are lower.
Related event: Memory Bandwidth Caps LLM Decoding: A Simple Token/s Formula(2 posts)→
More from Infra
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Decision Models' routing classifier hits AntSeed, cutting inference cost to 24% of always-Opus — airesearch12 · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20
- a16z's Ben Horowitz says AI buildout isn't the 90s bubble — an investor pushes back — henloitsjoyce · 2026-10-20
- Micron and Nvidia drive ~36% of S&P 500's Q3 earnings growth — Beth_Kindig · 2026-10-20
- NVIDIA lands in Europe: Jensen Huang and German chancellor discuss industrial AI at GTC Berlin — nvidia · 2026-10-20