SageMaker prefix-aware routing cuts P50 TTFT by up to 77% via warm KV caches
AWS ML Blog · rss · 2026-09-11
Amazon SageMaker Inference launched PREFIXAWARE routing: it inspects the start of each request and consistently sends requests sharing a prefix to the same instance, so KV caches actually stay warm. Benchmarked on Llama 3.1 70B across 7 p5.48xlarge instances with vLLM, long-context workloads (8,000-token shared prefixes) saw P50 TTFT drop 71–77%, throughput up 15–16%, and KV cache hit rates rise from 25% to 82%; short-context gains were smaller (P50 down 13–16%).
Mechanics and safeguards:
- No tagging needed; the endpoint routes on payload content directly.
- Overload protection: once the target hits the ConcurrencyThreshold (1–1024), traffic spills to less busy instances, trading one cache miss.
- Scale stability: adding/removing instances shifts only a small fraction of traffic, so caches aren't invalidated.
PrefixLength ranges 1024–65536 (bytes for the native API, characters for the OpenAI-compatible API) and should cover the shared prefix plus enough unique content to spread load. Routing overhead is just 1.3–1.9ms. Best-fit patterns: RAG over shared documents, multi-turn conversations, templated instruction bots, and code completion.
More from Infra
- k3 Report Section Confirms Millions of Concurrent Sandboxes in Its RL Training Run — stochasticchasm · 2026-09-11
- k3 RL Run Reportedly Used ~50M Sandboxes With Millions Concurrent, Checkpoints Merged Across Scaffolds — stochasticchasm · 2026-09-11
- Pentagon in talks to lend roughly $5 billion to AI cloud startup Fluidstack — vitaliychiley · 2026-09-11
- Eric Schmidt: AI may hit a money wall before a power wall — $1T capital needed — rohanpaul_ai · 2026-09-11
- SpaceX signs another AI compute deal: $1.11B per month, on track for $100B ARR — NinaDSchick · 2026-09-11
- Carmack: Jetson Thor's 128GB at 273GB/s is over-provisioned for real-time robotics — ID_AA_Carmack · 2026-09-11