AURA: Agentic Diagnosis and Refinement for Production Recommender Systems at Scale
SungGeun Kim, Abhinav Narain, Daniel Nemirovsky
cs.IR, cs.AI, cs.LG
2026-09-15
AURA reads ~100k Disney sessions per platform and lifts rubrics to 23/25 and 24/25. Genre mismatch is ~half of BAD cases; code fixes stay within ±0.1% offline.
AUC, NDCG, and coverage tell a recommender team whether the average moved. They do not say whether kids' profiles got horror promos, sports fans got reruns, or shelf dedup broke. Aggregate scores also wash out slice-level harm. A/B tests can identify causes and are expensive, slow, and risky on high-impact surfaces. Engineers fill the gap by writing cohort queries and watching a handful of sessions, which does not scale.
AURA (Agentic Understanding and Refinement of recommender Algorithms) chains that workflow: read production sessions, taxonomize failures, ground hypotheses in the ranker's own code, and open a PR. The work comes from The Walt Disney Company, for the GenAI e-commerce workshop at RecSys 2026.
Four stages. Session selection mixes random or stratified samples, domain SQL, and parallel screening agents, then dedups and majority-votes the cohorts. Diagnosis runs hierarchical aggregation over hundreds of thousands to millions of sessions. The deployed funnel has six steps: batched classification, cross-batch consolidation, sampling by size or safety, per-category analysis, verification, synthesis. Hypothesis and proposal reads ranker code, features, training, and schemas, and writes a root-cause hypothesis plus change ideas. Code implementation currently stops at engineer-reviewed pull requests; automated training and A/B remain dashed future work.
Cost is routed: high-volume classification goes to a light model (Gemini 3 Flash in the paper), while interpretation, verification, and coding use frontier models. Every LLM boundary has a programmatic check: session IDs must exist in the input, file paths must exist in the repo. Agents are read-only on analytical lakes. Code edits happen on a sandboxed clone. A named engineer owns any ship. Dismissal is a first-class outcome.
Two streaming platforms. Family-oriented Platform A: 18,901 BAD sessions out of 96,801 (about 19.5%). General-audience Platform B: 4,154 BAD out of 101,594 (about 4.1%). The rates are not a quality ranking; prompts are calibrated per platform and B's prompt changed mid-stream.
A three-model panel (Claude Opus 4.6, GPT-5.5, Gemini 3.1 Pro) on a stratified 200-session sample majority-agreed with the upstream judge on 96.0% of A (192/200) and 87.6% of B (176/201). Diagnostic quality uses 25 binary rubric criteria scored by two LLM judges on real outputs. After six architecture fixes plus prompt edits, A moved 15/25 to 23/25 and B 17/25 to 24/25. Most of the lift was architectural: a severity-alignment failure was the template never receiving the category's session share.
A locked vocabulary beat open-ended tags. B went from 30-plus overlapping labels to a 13-category closed list, with consolidation reduced to one rename. A's 12 emergent tags collapsed to 8. In a representative run, genre-preference mismatch covered 48.3% of A's BAD sessions (9,135) and 46.0% of B's (1,911); sub-genre mismatch was 34.1% on A. Safety-relevant low-count categories survive via a coverage rule. A also flagged 250 sessions as the upstream judge being wrong.
For B's 1,911 genre-mismatch sessions, the remediation stage proposed a user-genre × candidate-genre cross feature and candidate-aware attention. Across three independently trained days, offline deltas versus a production clone stayed inside ±0.1%: Click AUC at most +0.06%, NDCG@10 between -0.01% and +0.05%. The finding was not redeemed by those edits.
End-to-end cost was $349.26 on A and $252.85 on B, of which upstream per-session judges were 91.9% and 99.0% (about $321 and $250). AURA's own aggregate diagnosis was $28.28 and $2.64. A baseline B run had the analysis stage invent 904 plausible UUIDs; sanitization now drops any ID missing from the input.
If a team already pays for per-session LLM judges, the expensive layer is that judge. Building a failure vocabulary, hypotheses, and PRs on top is cheap at the margin. AURA turns stories that averages hide into session IDs an engineer can open, and it ported across two catalogs and two models as configuration. Today it is a high-throughput diagnosis-and-proposal system. The offline gate rejecting two genre fixes an engineer might have tried first is useful in itself: one live A/B not spent.
There is no held-out human audit of diagnostic correctness and no external baseline; both are listed as future work. Closed-loop training and A/B still sit behind the engineer checkpoint. The offline repairs did not move ranking on the diagnosed cohort. Rubrics and judges are LLMs, and Gemini appears upstream and downstream, which can inflate agreement. Severity is a prevalence tier, not per-event harm. Platform A tags are non-exclusive, so shares sum past 100%. The e-commerce transfer argument is a configuration table, not retail data.