2026-10-01
Bedrock's WLM serves millions of requests/min in 30+ regions by decoupling ingress from execution and preferring bounded inconsistency to global sync.
Foundation-model inference platforms sit on scarce accelerators that are not evenly stocked across regions. Token burn varies by orders of magnitude and is only revealed as generation proceeds. The same pool must serve chat that cares about latency and batch jobs that care about throughput, with SLAs that conflict.
This is an industry paper on the Workload Management System (WLM) already running in Amazon Bedrock. The abstract's scale claim is millions of inference requests per minute across more than 30 regions. ACM's Digital Library returned a Cloudflare challenge; the paper appeared on 2026-09-28 and no preprint or author PDF showed up. What follows is the Crossref/OpenAlex abstract only. No evaluation number is invented.
Two design principles.
First, split request ingress from execution and form a logical execution pool inside the allowed geographic and model-placement scope, with prompt-cache stability as a first-class routing constraint. KV / prompt-cache hits need similar prefixes on the same machines. Routing on spare flops alone shatters that locality, and both time-to-first-token and accelerator utilization suffer. Treating cache stability as a routing input is an admission that generative traffic is much more local than a stateless API.
Second, prefer bounded inconsistency plus defense in depth over synchronous global coordination. Cross-region synchronous admission would put the request path on the slowest consensus round. WLM admits and routes from local or cached state; quota consumption is reconciled across regions on a timer; routing weights are refreshed off the request path from regional capacity and health.
Five named subsystems share the work: admission control, traffic classification, cross-region routing, request scheduling, and orchestration. Classification likely separates latency versus throughput goals before routing and scheduling; orchestration maps the logical pool onto real accelerators. The abstract does not describe algorithms, queue structure, or failover.
The evaluation mixes production observation, benchmark measurements, and a controlled load test of bounded-staleness admission against synchronous admission. Reported axes are accelerator utilization, cache stability, request latency, priority behavior, throughput, request-path overhead, quota-enforcement accuracy, and behavior under selected failures.
The abstract contains no numbers. No utilization percentage, no latency percentile, no throughput delta versus synchronous admission, no recovery time. The PDF runs pages 52–64 (13 pages); those figures should live in the results section. They are not available in this pass.
What is clear is the experimental bet: bounded inconsistency is treated as a measurable choice, with synchronous admission as the foil. The authors think globally consistent quotas and weights are the wrong default at this scale, and they are willing to argue from production traffic.
Most public LLM-serving papers stop at a cluster: scheduling, prefill/decode split, prefix-aware routing. Bedrock has to do that across models, regions, and SLAs, without breaking prompt cache at the routing layer. Millions of requests per minute in 30-plus regions is a rare production disclosure.
Two takeaways travel. Decoupling ingress from execution is how a geo-fenced accelerator fleet becomes one pool. Keeping quota reconciliation and routing-weight updates off the hot path is the same class of trade as stale reads in a database, applied to GPU quota. Prompt-cache stability as a first-class constraint means cache hits are expensive enough to steer cross-region routing, not a later optimization.
The binding constraint on this write-up is missing full text. The abstract names five subsystems and eight metric families, and gives no baseline, ablation, or failure set. Whether WLM covers only Bedrock's serverless path, how reserved throughput is handled, and how provider capacity contracts enter the pool are all unstated.
What "bounded inconsistency" means for quotas stays literal: a region may oversell or reject too hard in a short window, then the reconcilers catch up. How many defense-in-depth layers exist is unknown. If traffic classification is only a priority tag, it may or may not match the usual prefill/decode split; the abstract does not say. The license is CC BY-NC-ND, so the PDF should be downloadable; the block is ACM's bot wall, not a paywall.