How Salesforce serves 30+ regions with SageMaker cross-Region inference routing
AWS ML Blog · rss · 2026-11-19
- Salesforce's Agentforce Foundation team serves models across 30+ global regions and hundreds of AWS accounts. Per-region capacity provisioning caused fragmentation: idle instances in some regions, shortages in others. Their hand-rolled Internal Gateway failover lacked capacity signals and sent cross-region traffic over public networking, with cascading-failure risk during incidents.
- How the router works: apps invoke a regional SageMaker inference router instead of endpoints; each model is a routing target listing destination endpoints by ARN, which can live in other regions/accounts. The router prefers its local region, scores destinations by recent request outcomes (5xx, throttles, timeouts), shifts traffic within a minute of persistent errors, and returns traffic slowly after recovery. Client errors and model latency don't trigger failover; a throttled request is retried once elsewhere.
- Salesforce integration: the router call sits in the Internal Gateway's egress path after the data-residency check; residency-tagged requests are pinned to an in-region-only routing target. The gateway logs InvokedRoutedResourceArn and RoutingReason as first-class telemetry, joined against CloudWatch metrics.
- Gateway timeouts account for cross-region round trips and add no extra retries, avoiding attempt multiplication during incidents.
More from Infra
- vLLM's DSpark for Kimi K3.1 Pro: 30% longer acceptance, bounded draft KV at 1M context — AccBalanced · 2026-11-19
- Costco sees panic buying of generators after string of tech outages, 21 of 26 sold out — harris_edouard · 2026-11-19
- Starlink direct-to-cell goes live in Mongolia with free first month — XFreeze · 2026-11-19
- SpaceX unveils $50B, 6GW Arizona compute campus with 24 GWh grid-battery strategy — JOBhakdi · 2026-11-19
- CoreWeave corrects debt rate to 15.5% as analyst presses Blackwell-funded payoff plans — Svwang1 · 2026-11-19
- tinygrad lists $89,000 tinybox with 768GB VRAM: four refurbished AMD MI300X GPUs — viksit · 2026-11-19