How to Estimate tokens/sec on Your Hardware: The VRAM Bandwidth Formula
Pyrolistical · reddit · 2026-09-05
A Reddit user shares a practical method for estimating local LLM decoding speed. Key insight: the decoding bottleneck is memory bandwidth, not compute — every generated token requires reading all model weights plus the KV cache from VRAM.
Basic formula (dense models): TG/s = VRAM GB/s ÷ model weights GB
Worked example: Qwen3.8 27B Q4KXL weighs 16.8 GB (excluding MTP layer and input embedding table); on an AMD Radeon AI PRO R9700 (637 GB/s), the theoretical max is 637/16.8 ≈ 38 TG/s. The author measures 29 TG/s with llama.cpp — 76% of ideal.
Full formula with KV cache: TG/s = VRAM GB/s ÷ (weights GB + KV cache GB/token × context length). Qwen3.8 27B's BF16 KV cache costs 64 KB per token. Rearranged, this yields a curve of TG/s-per-GB/s vs context size. For a 5090 (1.8 TB/s): 106 TG/s max at short context, dropping to 53 TG/s at 256k context.
Caveats: assumes everything fits in VRAM; simplified formula is dense-only; speculative decoding stacks on top; these are theoretical ceilings — real-world numbers are lower.
Related event: Memory Bandwidth Caps LLM Decoding: A Simple Token/s Formula(2 posts)→
More from Infra
- AMD, Cisco and Saudi Arabia's HUMAIN deploy MI335X GPUs, planning up to 250MW — Beth_Kindig · 2026-09-05
- Reef launches inference-native infra that serves self-improving agents without downtime — pliang279 · 2026-09-05
- From 1D to 2D int8 kernels: a hands-on GPU internals learning path — goyal__pramod · 2026-09-05
- Google DeepMind Publishes Free Book on Scaling LLMs Across TPUs and GPUs — goyal__pramod · 2026-09-05
- Prime Super Flash MoE: 1.2x BF16 and 1.6x MXFP8 speedups over upstream on B200 — retr0jirachi · 2026-09-05
- Google Cloud proposes Cloud Run sandboxes to fix coding agents' repair loop — rseroter · 2026-09-05