Why faster GPUs don't yield more tokens: H100 needs 295 ops per byte fetched

AccBalanced · x · 2026-11-16

A long-form explainer argues compute ratings rarely predict token throughput: arithmetic units mostly wait on memory. The H100 does 989 TFLOPS at 16-bit against 3.35 TB/s of memory bandwidth — an arithmetic intensity of 295 ops per byte, meaning compute only stays fed at very high intensity. Memory bandwidth also grows far slower than FLOPS each generation, which is why quantization, speculative decoding, and continuous batching exist. Alongside the article, the author open-sourced time-to-first-token (1.2k stars), a 10-week, 30-minutes-a-day roadmap covering vLLM, SGLang, quantization, speculative decoding, and benchmarking, built around shipping one real inference service.

Original post →

More from coding & agent

coding & agent channel →