A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, Juncheng Yang
cs.AI
2026-07-04
A year of 6.12B LLM requests: median output falls under 100 tokens, prefix hits are bimodal with 99% of repeats in 15 min, and FIFO matches SOTA caches.
Serving-system papers need traces you can replay. Public artifacts have been short, sampled, stripped of user identifiers, or replaced by workload generators. ServeGen already sits at 3.54 billion requests over four months and 12 models, and that is still too narrow to watch model turnover over a year, users switching models, or the fight between prefix-cache locality and load balance.
This paper logs, and says it will release, a year of unsampled request-level production traffic from CompanyX, a serverless LLM inference platform (two authors are at Chutes): 11 April 2025 to 12 April 2026, 6.122 billion requests, 314,970 users, 9,174 models (3,922 public, 5,252 private), 875,921 serving instances. About 3.58×10¹³ input tokens and 2.52×10¹² output tokens. Fields include anonymized userid, instanceid, input/output length, TTFT, and cachedtokens.
The analysis stacks five views: aggregate load, user/model structure, year-long evolution, prefix caching, and routing simulation. Unless noted, it uses the full year of text APIs, streaming and non-streaming chat and completion. Three high-traffic models illustrate distinct profiles: DeepSeek-V3.2 for chat and role-play, DeepSeek-R1 for long reasoning outputs, MiniMax-M2.5 for coding and agents.
There are no prompts and no session IDs. Sessions are reconstructed inside each user from the last K=10 requests: same model and endpoint, parent input+output strictly nested in the child input, unique tightest residual. The reconstruction ignores production routing and logged cache hits, so the simulator estimates reuse that is in the workload rather than reuse created by the live router. Eviction is replayed with libCacheSim. Routing then places those sessions on N instances with local LRU, comparing round-robin, load-first, sticky, and cache-first.
Demand is non-stationary, and request count is not a proxy for work. Daily requests rise through mid-2025, peak in late summer/fall, then fall; daily tokens drop more sharply. Active models go from under 100 per day to more than 400. Chat grows, completion and streaming shrink. Weekday UTC afternoons and evenings are the busy window. Requests are prompt-heavy: inputs roughly 10²–10⁵ tokens, outputs mostly 10¹–10³, most outputs shorter than inputs. End-to-end latency is decode-dominated. Output length drives duration, input length drives TTFT, and cache hits cut TTFT most on long prompts.
Similar request volume hides very different cost. Many high-traffic models are used by few users; most users touch few models, with a tail of high-volume, many-model users. Most models have IAT CV>1 and positive lag-1 autocorrelation, so busy and quiet spells persist. Dominant models turn over: DeepSeek-V3-0324 and R1 lead early, Qwen3-32B and DeepSeek-V3.2 later, and the "Others" band widens. Global median input stays in the thousands of tokens all year; median output falls from a few hundred early to under 100 by the end. Later user cohorts send longer inputs and spread more.
Logged cachedtokens exist only for the last two months. Request-level token hit ratios are bimodal: near zero or near the full prefix, little in between. Short prompts skew high-reuse, long prompts skew low. MiniMax-M2.5 has more intermediate-to-high hits; the two DeepSeek models have more zero-hit requests. For consecutive (user, model) arrivals, about half return within 0.1s, about 80% within 10s, and 99% within 15 minutes. In single-cache replay, FIFO and LRU match or beat ARC, Sieve, GDSF, and LRB, the usual web/storage SOTA. DeepSeek-V3.2 stays below a size-aware Belady oracle even at large caches; MiniMax approaches the oracle at moderate sizes.
In production, the balancer keeps a heavy user on few instances when load is low and spreads that user when load is high, replicating KV. A 100K-token MiniMax context is about 27GB of KV, roughly 2.7s to move one way on a 10GB/s link. In simulation, cache-first turns larger per-instance caches into higher hit ratios. Round-robin and load-first scatter sessions, so extra cache barely helps. Cache-first pays about 5–7% max/mean token/s imbalance. Sticky is worse by orders of magnitude and is omitted from that panel.
ShareGPT plus synthetic arrivals will miss model turnover, shrinking outputs, users hopping models, and the all-or-nothing hit pattern. On this trace, complicated CDN/storage eviction often loses to FIFO. Autoscaling on QPS alone drifts from actual GPU work, which tracks tokens. Routing cannot treat balance and caching as separate knobs: spreading copies KV, sticking unbalances load. The 5–7% figure assumes many one-off requests that can still fill idle replicas. Stickier multi-turn traffic will cost more.
The trace is the main deliverable. A year of unsampled requests with anonymized users, instances, TTFT, and cached tokens lets other people replay arrival, reuse, and routing instead of inventing them.
One serverless platform, more than half private user-deployed models, so closed-lab API traffic need not look like this. Cache logs cover only the last two months. Session reconstruction is a heuristic (K=10, token nesting) with no prompt text to check. The simulator assumes those sessions and local LRU; it is not a production A/B. The remaining gap to Belady says current eviction leaves reuse on the table, and the paper does not propose a replacement. Cross-user KV hits are treated as rare and not re-measured here. The manuscript says the trace will be released; a public URL is not in the draft.