2026-09-02
DynamoLLM retunes LLM serving pools, parallelism, and GPU clocks, cutting 53% energy, 38% carbon, and 61% cost on week-long traces while meeting latency SLOs.
LLM serving clusters are typically sized for peak and then left at maximum GPU frequency and maximum tensor parallelism. Power and idle draw follow the peak, not the work. Requests are not one workload. Prefill is compute-heavy; decode is memory-heavy. Short chats and long contexts want different machines. On two Azure week-long traces, Conversation peaks at 1.7× its mean and 3.3× its valley; Coding peaks at 2.8× mean and 34.6× valley. The same max-performance setup burns idle watts at night and on weekends.
Classic datacenter power managers were built for CPU jobs. Reconfiguring LLM serving is expensive. Spinning up an 8×H100 instance takes 6-8 minutes. Setting frequency through nvidia-smi takes 50-80 ms, while a decode step is 20-30 ms, so retuning every iteration wrecks throughput. The missing piece is a controller that picks configurations automatically and hides the switching cost.
DynamoLLM is an energy manager for LLM inference from UIUC and Azure Research, built on vLLM and published at HPCA 2025. It controls three knobs: instance count, tensor parallelism (TP2/TP4/TP8), and GPU frequency (800-1980 MHz in 200 MHz steps).
Before serving, it profiles energy and TTFT/TBT across lengths, loads, parallelism, and frequency. Profiles are reused across services that share a model. Requests fall into nine input/output length buckets using the 33rd/66th/100th percentiles of a Conversation trace. The SLO is 5× an isolated request: TTFT 250/400/2000 ms for short/medium/long, TBT 100 ms for all. A BERT proxy predicts output length; input length is known on arrival.
Three controllers run at different timescales. About every 30 minutes the cluster manager sizes pools from a load forecast, assuming TP8 at 1980 MHz as the capacity floor, and dumps leftover load into the next-longer pool so every pool is not over-provisioned for its own peak. About every 5 minutes the pool manager picks a TP split under a fixed GPU budget and subtracts resharding cost from the energy gain. About every 5 seconds the instance manager picks the cheapest frequency that still meets the SLO.
Switching cost is treated as a first-class input. New VMs use in-cluster weight caches, engine snapshots, and proactive boot before the epoch. Resharding uses bipartite maximum-weight matching to keep shards in place and NVLink for the rest; TP4 to TP8 is about 50 ms (one eighth of the weights), TP4 to TP2 about 100 ms. Old and new instances overlap when memory allows, so engine sync of hundreds of milliseconds to a few seconds is not downtime. Frequency changes keep nvidia-smi resident and run the controller privileged, skipping a user-kernel round trip.
Mis-prediction has a fallback. Underestimated length that fills a queue first reorders near-deadline requests, then jumps to max frequency, then re-steers unstarted work inside the pool; if that fails, long-waiting requests are dropped so the frontend can retry elsewhere. Overestimated length only lands in a more expensive pool; latency usually holds.
Hardware is 8×H100 servers. Headline numbers use Llama2-70B. SinglePool, the production-style baseline, puts every request in one TP8 max-frequency pool sized at 12 servers for the peak. Ablations add pools only, scale instances only, change TP only, or change frequency only.
On a one-hour public Conversation trace, versus SinglePool:
| System | Energy | Notes |
| MultiPool | +20% | Separate pools, still max performance, more machines |
| ScaleInst | −4.1% | Instance scaling; slow boots inflate tail latency |
| ScaleShard | −7% | TP only |
| ScaleFreq | −19% | Frequency only |
| DynamoLLM | −35% | All three knobs |
P99 TTFT/TBT drop 5.3%/11.1% versus SinglePool; P50 rises 11.4%/7.6% because slack is spent on cheaper configs. Cluster power P50/P99 fall 43%/9%.
A 24-hour production trace on 11 H100s saves 42% energy. A week-long discrete-event simulation saves 47% on Conversation and 56% on Coding. The abstract's 53% is a service-level headline in the same band as the week-long pair, and higher than the 35% one-hour cluster run.
Operational carbon is energy times CAISO carbon intensity: 5 t/week for SinglePool versus 3.1 t/week for DynamoLLM, a 38% cut. Mean GPU servers over the week fall from 40 to 24.6, about $1362.7/hour at then-current GPU VM prices. Energy cost falls by up to 56%, only about $4.4/hour. The 61% customer-cost number is mostly fewer GPUs.
A 40% output-length error adds 13% energy and 7.3% TTFT. Versus SinglePool, savings are 51%/40%/23.4% at low/medium/high load. Nine pools is the design point; 12 or 16 fragment, 2 or 4 cannot specialize.
Inference already buys GPUs for the peak. Idle machines and max clocks are the two default wastes. DynamoLLM turns pooling, parallelism, and frequency into a platform loop, and it charges resharding and scale-out delay against the energy win before it switches. For a vLLM fleet, the portable pieces are length-based pools, lower clocks at low load, and an NVLink cost check before changing TP.
Do not paste the 61% onto another bill. Energy savings are tiny next to GPU rent. Carbon is operational only, no manufacturing.
There is no standalone limitations section. Main plots are Llama2-70B; other models are said to follow the same trend. Only single-node tensor parallelism is implemented; pipeline parallelism is mentioned and dropped. Output length has to be predicted, and the last emergency move drops requests. Frequency control wants privilege and a resident nvidia-smi, which tenant clusters may not grant.
53%, 38%, and 61% are abstract figures. The numbers you can read off a real GPU cluster are −35% in one hour and −42% in one day. The week-long cuts are simulated. Carbon intensity is shown for CAISO only. Profiling sweeps TP and frequency; a new model or a looser 10×/20× SLO needs a new sweep.