How LLM Inference Bottlenecks Evolved: From Weights to KV Reads to HBM Capacity

YouJiacheng outlines how LLM inference bottlenecks evolved from weight loading to KV cache reads as long-context workloads grew, and now to HBM capacity, noting speculative decoding cannot change the bottleneck's nature, which is determined by model and workload.

2026-09-11 ~ 2026-09-11 · 2 related posts

• LLM inference bottlenecks: weight loading gave way to KV reads as contexts grew — YouJiacheng · 2026-09-11
• Inference Bottleneck Evolution: HBM Capacity Now Limits Batch Size, Not KV Reads — YouJiacheng · 2026-09-11