AWS launches HyperPod Inference Gateway: GPU-aware routing cuts first-token latency by up to 82%

AWS ML Blog · rss · 2026-09-18

AWS announced SageMaker HyperPod Inference Gateway, a Kubernetes-native, GPU-aware routing system installed as a single EKS managed addon with zero application changes. Default round-robin load balancers have no visibility into GPU state (KV cache saturation, LoRA adapter residency, queue depth), pushing first-token latency past 4s during bursts; in AWS tests, 4.4s dropped to under 800ms, up to an 82% reduction.

The two-tier design: Tier 1 is a per-cluster gateway with Envoy Gateway (TLS termination), a Body-Based Router (multi-model routing via the model field), and an Endpoint Picker that scores pods on KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit rate, and running requests using Prometheus metrics, with configurable weights. Tier 2, a Global Inference Router with cross-cluster failover and cost-aware traffic shaping, is coming soon.

Setup is three steps: install the addon via aws eks create-addon, label model pods, and apply an InferenceGatewayConfig CR to expose a standard OpenAI-compatible endpoint. It natively supports LoRA affinity routing (avoiding adapter swap latency), graceful degradation and self-healing (stale pods excluded, 429 with Retry-After on pool exhaustion, 35s cluster failover), and layered observability. Benchmarks span four models from 8B to 235B on H100 and A10G, with the biggest gains in mixed-GPU-generation fleets and bursty traffic.

Original post →

AWS launches HyperPod Inference Gateway: GPU-aware routing cuts first-token latency by up to 82%

More from Infra