AWS Tiered KV Cache Architecture Boosts LLM Inference TTFT by 2.7x
AWS ML Blog · rss · 2026-08-12
AWS introduced a tiered KV cache architecture on Amazon SageMaker HyperPod to solve GPU memory bottlenecks and cold start issues in large model inference by sharing caches across nodes.
Three-Tier Cache Design
- L0 (GPU HBM): vLLM's native paged-attention layer with the lowest access latency.
- L1 (Host Memory): Catches evicted GPU blocks into local node DRAM via LMCache.
- L2 (Distributed NVMe Pool): Introduces Curvine, a lightweight distributed cache filesystem, to pool node-local NVMe drives. This allows different inference replicas to instantly reuse KV blocks.
Combined with HyperPod's prefix-aware or kv-aware intelligent routing, the architecture achieves up to a 100% cross-Pod cache hit rate. Benchmarks show a 2.7x improvement in Time-To-First-Token (TTFT) for 1,900-token prompts, with cross-node L2 read latency around 56 ms. This allows workloads previously requiring expensive P5 instances to run on cheaper G6e instances.
More from Infra
- Nvidia Is Speedrunning the Creation of a Synthetic Hyperscaler — firstadopter · 2026-08-12
- Expert: 6-Inch Wafers Won't Entirely Solve Optical Comm Scaling Challenges — BenBajarin · 2026-08-12
- AI Compute Boom Drives TL20 Tech Stocks Up 59% Year-to-Date — TiernanRayTech · 2026-08-12
- How Vercel Migrated Its Core Database Handling 6,000 Deployments Per Minute — evilrabbit_ · 2026-08-12
- Analyst: Server CPU Market Bracing for Unprecedented S-Curve Leap — BenBajarin · 2026-08-12
- ComfyUI Workflows Crawl on 128GB DGX Spark vs RTX 4090 — jungseungoh97 · 2026-08-12