AWS launches SageMaker HyperPod Global Inference Gateway for cross-region, single-endpoint routing
AWS ML Blog · rss · 2026-11-21
AWS introduced the SageMaker HyperPod Global Inference Gateway, a managed endpoint that routes inference across all registered HyperPod clusters, within and across regions. It scores healthy clusters on weighted signals (region proximity 0.40, queue depth 0.25, KV-cache headroom 0.20, error rate 0.15 by default), overflows requests to idle GPUs elsewhere during peaks, and fails over automatically — all over PrivateLink with SigV4/IAM auth. Setup: publish each cluster via the globalRouting block, then create a gateway with CreateGlobalInferenceGateway. Beta customer Bagelcode got cross-region failover across two regions through one endpoint without building its own routing layer.
More from Infra
- Hugging Face adds H200 inference endpoints at $5 per GPU-hour with one-click deploy recipes — lhoestq · 2026-11-21
- JapanFold on Tenstorrent silicon predicts 1M+ protein structures powering Anthropic design contest — DavidBennett__ · 2026-11-21
- DeepSeek-V4.1-Flash on one Apple M5 Ultra 512GB: steady 101 tok/s decode up to 256K context — HankYeomans · 2026-11-21
- Keysight Q4 orders jump 53% to $2.34B as AI testing demand still outruns supply — BenBajarin · 2026-11-21
- Micron CEO: customer demand over next five years exceeds what we could commit — Beth_Kindig · 2026-11-21
- 2026 WebAI Summit Posts 25 Talks, 8 Per Week, Keynoted by Google's Jason Mayes — jason_mayes · 2026-11-21