CORAL: An LLM-Native Harness for Production Recommender Systems
Muhammad Rafay Azhar, Yuhang Zhou, Gilbert Jiang, Yuchen Wang, Rahul Sharma, Matthew DeSousa, Jiayi Liu, Xin Guo, Lizhu Zhang, Xiangjun Fan
cs.CL
2026-09-02
CORAL is an LLM loop that retunes a live recommender under a budget: +0.16% sessions on video, and millions USD/year saved on a second surface with flat engagement.
A production recommender is a pile of hand-set knobs around retrieval, ranking, and serving: candidate budgets, ranker weights, cache and prefetch policy, per-segment treatments. Those knobs are not learned end-to-end with the models, and the best setting drifts as content, behavior, and upstream models move. Human online experiments chase the drift at the speed of engineering headcount. Low-signal and new users are easy to underserve because aggregate metrics hide them.
Most LLM-for-recsys work edits ranking, user memory, or offline development. Few systems put an agent on the live control surface and let it update itself from the measured effect of its last change.
CORAL (Constraint-Optimized Recommender via an Agentic Loop) writes continual tuning as a partially observed, non-stationary, constrained problem. Each cycle picks a configuration that maximizes the current objective (engagement, or its dual, cost) subject to a hard budget B. The policy is a general-purpose LLM. Weights do not update. Improvement is supposed to come from observations and memory in context.
The harness supplies three things the model cannot guarantee. Memory has three stores: raw per-unit stats, the model’s own natural-language assessments, and deployed configurations with attributed outcomes, with a default horizon of three cycles. Tools cover analysis, memory lookup, attribution, and a numerical optimizer: the agent proposes bounded per-unit moves, the optimizer projects onto the feasible set, and redistribution happens only when the proposal would overspend. The loop runs in a fixed order. Deployments used a 3-day cadence. A configuration stays live for a full cycle, then an A/B result is written back. Humans watch first; the intended path is to replace that watch with feasibility checks, step limits, and safety constraints.
Both case studies fill the same prompt template. On video, control units are retrieval sources and the action is a budget multiplier. On the second surface, control units are user segments and the action is a discrete serving treatment from light to heavy.
Retrieval-budget allocation. A zero-shot first round gained +0.13% watch time with no significant session move. The second round over-shifted and went neutral. The later deployed allocation, in a million-user A/B, delivered +0.16% video-viewing sessions and +0.15% watch time with no extra retrieval budget. Per-segment policies then raised sessions for new low-signal users by 0.23%, by moving budget from history-heavy sources toward content and current-context sources that still work when user logs are thin.
Serving-capacity allocation. The first round cut serving cost on a subset of segments, at millions of USD in annualized capacity. Reading that result, the agent widened the same treatment in round two and increased the savings by 44%, with engagement statistically unchanged.
LLM cost is charged per decision cycle, not per request. Each cycle makes about 8–10 calls of 1.5k–2k input tokens and 2.5k output tokens. A few cycles land around 10^6 tokens, tens of dollars at frontier prices.
This is the algorithm engineer’s recurring job (reallocating a bounded budget) handed to an agent, with online A/B evidence that one harness can sit on both sides of the engagement–efficiency frontier. Because the agent never runs per request, cost does not scale with QPS and stays small on a billion-user surface. A 3-day cycle is an order of magnitude faster than a human turning one knob for weeks. A 0.16% session lift is real on a large surface and small as a product story. What it shows is that the loop can keep moving budget, not that it rewrote recommendation quality in one shot.
The loop still runs under human supervision; stronger guardrails are listed as future work, which is another way of saying unsupervised operation is not ready. Both studies allocate bounded resources across components. Retrieval logic and the ranker itself are untouched. Evidence is expensive, setting-specific A/B tests; there is no standard offline protocol for this class of agent. Improvement is not monotonic: round two overcorrected. Cadence k and memory m were defaults, not a seasonal search. Product names and absolute QPS or dollar figures are omitted, so outsiders cannot reproduce the scale.