AWS ships 13 SageMaker inference updates in 2026, cutting cold-start latency up to 65%
AWS ML Blog · rss · 2026-09-19
The AWS ML Blog reviews the 13 inference capabilities Amazon SageMaker AI has shipped year-to-date in 2026, across two deployment paths: fully managed Endpoints and Kubernetes-native HyperPod Inference.
Key launches on the managed Endpoints path:
- Inference recommendations (April): automates what used to take 2-3 weeks of manual benchmarking across 1000+ instance/container combos, applying EAGLE 3.0 speculative decoding, kernel tuning, and tensor parallelism, validated with NVIDIA AIPerf multi-run confidence reporting. Demo showed 2x tokens/sec on GPT-OSS-20B at equal latency, free of charge.
- Capacity-aware instance pools (May): up to 5 prioritized instance types with automatic fallback at creation, scale-out, and scale-in, eliminating single-point-of-failure from GPU shortages.
- OpenAI-compatible APIs (May): an /openai/v1 path with streaming Chat Completions—migrate by changing the endpoint URL only; bearer-token auth replaces SigV4; live in 14 regions with vLLM/SGLang containers.
- Container caching (June): pre-pulled images cut end-to-end startup from 525s to 258s (-51%) in a Qwen3-8B test; early customers saw 38-65% improvements.
HyperPod Inference gained a simplified operator, tiered KV cache, data capture, disaggregated prefill/decode, and model caching. Together the releases target generative AI inference's core pain points: multi-minute cold starts, GPU capacity fragility, and missing token-level observability.
More from Infra
- Dev builds daily-driver AI coding environment on AWS Lambda MicroVMs with S3 persistence — blaizedsouza · 2026-09-19
- Running 35B–284B MoEs at 128k Context on a 12GB A2000: Measured, Plus What Didn't Matter — HomoAgens1 · 2026-09-19
- Joseph Suarez hits SOTA in 5 hours on consumer cards in a warehouse — yacineMTB · 2026-09-19
- MiMo RL livestream: infra restarts waste 33% of compute, over $400K lost — dustinvtran · 2026-09-19
- Why Jev-class models could become a near-free judgment primitive running on-device — signulll · 2026-09-19
- Pedro Domingos: Opposing Data Centers Means Keeping Your Country Stupid — pmddomingos · 2026-09-19