SageMaker prefix-aware routing cuts P50 TTFT by up to 77% via warm KV caches
AWS ML Blog · rss · 2026-09-11
Amazon SageMaker Inference launched PREFIXAWARE routing: it inspects the start of each request and consistently sends requests sharing a prefix to the same instance, so KV caches actually stay warm. Benchmarked on Llama 3.1 70B across 7 p5.48xlarge instances with vLLM, long-context workloads (8,000-token shared prefixes) saw P50 TTFT drop 71–77%, throughput up 15–16%, and KV cache hit rates rise from 25% to 82%; short-context gains were smaller (P50 down 13–16%).
Mechanics and safeguards:
- No tagging needed; the endpoint routes on payload content directly.
- Overload protection: once the target hits the ConcurrencyThreshold (1–1024), traffic spills to less busy instances, trading one cache miss.
- Scale stability: adding/removing instances shifts only a small fraction of traffic, so caches aren't invalidated.
PrefixLength ranges 1024–65536 (bytes for the native API, characters for the OpenAI-compatible API) and should cover the shared prefix plus enough unique content to spread load. Routing overhead is just 1.3–1.9ms. Best-fit patterns: RAG over shared documents, multi-turn conversations, templated instruction bots, and code completion.
More from Infra
- Meta approves nonvoting Class C shares as 2026 capex guidance hits $135B–$145B — thione · 2026-10-26
- Engineer with thousands of fabricated devices pours cold water on photonic crystal modulator yields — jwt0625 · 2026-10-26
- Anthropic S-1 reveals 48 cents of compute cost per dollar of Q2 revenue — nmasc_ · 2026-10-26
- Persistent AI joins UR network to route private AI inference with encrypted, IP-hidden traffic — 0xSammy · 2026-10-26
- Anthropic's disclosed compute commitments: $114.5B Google, $161.2B Broadcom top the list — BenBajarin · 2026-10-26
- SpaceX strikes $40B deal with Apollo Global to lease AI chips and data center hardware — Polymarket · 2026-10-26