AWS launches SageMaker HyperPod Global Inference Gateway for cross-region, single-endpoint routing

AWS ML Blog · rss · 2026-11-21

AWS introduced the SageMaker HyperPod Global Inference Gateway, a managed endpoint that routes inference across all registered HyperPod clusters, within and across regions. It scores healthy clusters on weighted signals (region proximity 0.40, queue depth 0.25, KV-cache headroom 0.20, error rate 0.15 by default), overflows requests to idle GPUs elsewhere during peaks, and fails over automatically — all over PrivateLink with SigV4/IAM auth. Setup: publish each cluster via the globalRouting block, then create a gateway with CreateGlobalInferenceGateway. Beta customer Bagelcode got cross-region failover across two regions through one endpoint without building its own routing layer.

Original post →

More from Infra

Infra channel →