Why faster GPUs don't yield more tokens: H100 needs 295 ops per byte fetched
AccBalanced · x · 2026-11-16
A long-form explainer argues compute ratings rarely predict token throughput: arithmetic units mostly wait on memory. The H100 does 989 TFLOPS at 16-bit against 3.35 TB/s of memory bandwidth — an arithmetic intensity of 295 ops per byte, meaning compute only stays fed at very high intensity. Memory bandwidth also grows far slower than FLOPS each generation, which is why quantization, speculative decoding, and continuous batching exist. Alongside the article, the author open-sourced time-to-first-token (1.2k stars), a 10-week, 30-minutes-a-day roadmap covering vLLM, SGLang, quantization, speculative decoding, and benchmarking, built around shipping one real inference service.
More from coding & agent
- CAFE News Tracker Rounds Up XState 6 Betas, Cloudflare Cluster in Effect, Foldkit Full-Time — samgoodwin89 · 2026-11-16
- RatTube Launches Curated S-to-F Tier List of Effect/Alchemy Learning Videos — samgoodwin89 · 2026-11-16
- Developer teases game demo built with Fable 5.2, live Nov 18 — ChrisGPT · 2026-11-16
- Solo dev ships iPad math game for kids in ~2 days using Antigravity agent swarms — jacalulu · 2026-11-16
- Open-source MCP server lets Claude build a playable CS2 town from phone photos — SilentPrinciple8997 · 2026-11-16
- Redditor builds "Fish Report Tuesdays" on a 100% AI-made custom game engine — JynsRealityIsBroken · 2026-11-16