Inference Bottleneck Evolution: HBM Capacity Now Limits Batch Size, Not KV Reads
YouJiacheng · x · 2026-09-11
YouJiacheng outlines three stages of LLM inference bottlenecks: early workloads were weight-loading bound; long-context workloads made KV cache reads the bottleneck; and now, with DSA and even longer contexts, batch size is constrained by HBM capacity instead. He argues this is driven by the nature of the model and workload, not something speculative decoding can change.
More from Infra
- AWS tutorial: fleet-wide edge CV model lifecycle with SageMaker SDK v3, ~$0.07 per run — AWS ML Blog · 2026-10-27
- Qwen3.8-27B hits 40 tok/s on Jetson Thor with INT4 + MTP, accuracy beats FP16 — No-Dragonfly6246 · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27
- Supabase founders on rebuilding the backend for a billion AI-built apps — Latent Space · 2026-10-27
- AW-0 runs at 230ms on a single RTX 5090, 2.4x faster than autoregressive code planning — trevordarrell · 2026-10-27
- Site tracks US city council topics: data centers rank only behind pickleball courts — zakkohane · 2026-10-27