SageMaker prefix-aware routing cuts P50 TTFT by up to 77% via warm KV caches
AWS ML Blog · rss · 2026-09-11
Amazon SageMaker Inference launched PREFIXAWARE routing: it inspects the start of each request and consistently sends requests sharing a prefix to the same instance, so KV caches actually stay warm. Benchmarked on Llama 3.1 70B across 7 p5.48xlarge instances with vLLM, long-context workloads (8,000-token shared prefixes) saw P50 TTFT drop 71–77%, throughput up 15–16%, and KV cache hit rates rise from 25% to 82%; short-context gains were smaller (P50 down 13–16%).
Mechanics and safeguards:
- No tagging needed; the endpoint routes on payload content directly.
- Overload protection: once the target hits the ConcurrencyThreshold (1–1024), traffic spills to less busy instances, trading one cache miss.
- Scale stability: adding/removing instances shifts only a small fraction of traffic, so caches aren't invalidated.
PrefixLength ranges 1024–65536 (bytes for the native API, characters for the OpenAI-compatible API) and should cover the shared prefix plus enough unique content to spread load. Routing overhead is just 1.3–1.9ms. Best-fit patterns: RAG over shared documents, multi-turn conversations, templated instruction bots, and code completion.
More from Infra
- Anthropic's October ARR likely went vertical after token price cuts — GavinSBaker · 2026-10-26
- Anthropic ARR goes vertical after price cuts: tokens up 4x, $100B+ ARR back in play — GavinSBaker · 2026-10-26
- Alex Banks argues power caps GPU supply, real risk is demand cooling first — thealexbanks · 2026-10-26
- Analyst: Ant has the most fungible AI workload of any model provider today — BenBajarin · 2026-10-26
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26