Auto-RecSys: Harnessing Autonomous Research Agents for Industry-Scale Recommender System
Ming Li, Dai Li, Xuying Ning, Bo Sun, Rui Li, Yi Zhang, Silvia Gong, Xuan Cao, Rui Li, Cornelia Carapcea, Qunshu Zhang, Zhigang Wang, Yinglong Xia, Andy Wang
cs.CL
2026-09-10
Auto-RecSys is Meta's harness for multi-day recsys work: parallel runs, cross-server recovery, playbooks. Across 31 iterations, operational fixes fell from 4.0 to 0.5 per run.
Auto-research agents already chain hypothesis, code, experiment, and revision on small problems. Industry recommenders do not run on that clock. One training job takes hours to days. Configurations span thousands of lines. Jobs die on preemption, bad checkpoints, and dependency skew. Serial iteration stalls. Researchers spend the cycle on execution, not on ideas.
The contrast is explicit. Small-scale auto-research has minute-to-hour feedback and local reruns. Industry recsys needs recovery across sessions, parallel exploration, and a plan for failure as the default. The missing piece is not a better paper-writing agent. It is a harness that survives a multi-day GPU job.
Auto-RecSys tracks each idea with its own state machine: ideate, implement, validate, train, analyze, with a debug branch on training failure. State is per-idea, so several ideas on one model can sit at different stages on different machines without clobbering each other.
Three harness patterns. Distributed async execution never assumes the same session or server will still be there when training finishes. A shared memory store holds state, playbooks, and trajectory logs; a new session reads the global registry and resumes. Cognitive-procedural separation puts planning in markdown skill files and leaves state transitions, APIs, and file writes to deterministic scripts, so the LLM cannot corrupt JSON state.
A dual loop then improves the system. The execution loop distills trajectories into a per-model playbook: key files, validation commands, submission parameters, dead ends, and working pipelines. The idea loop appends verdicts to a history log so later ideation can deduplicate, combine partial wins, and drop repeatedly failed families. Playbook structure transfers across models in one shot; a new model still needs an interactive first pass to fill the slots.
Human-in-the-loop has two modes. Interactive pauses at idea pick, code review, train submit, and results. Autonomous skips the gates. Autonomous mode is meant to turn on after the playbook matures.
There is no report of recommendation-metric gains. The measured outputs are human time and execution reliability. Manually taking a comparable idea from proposal to analysis costs hours to days of attention. In interactive mode the human reviews ideas and diffs; the rest is delegated, and hands-on time falls to minutes. The same attention that used to cover one idea now covers more than a dozen, which is what makes a parallel portfolio feasible.
On one representative model they logged 31 unique iterations. During stabilization (iterations 5–20) major operational fixes fell from 4.0 to 1.3 per run. At iteration 21 the baseline switched to graph compilation plus a new embedding module and task adapters, which changed the submit stack; five straight iterations needed recovery. Iterations 26–31 then hit 0.5 fixes per run, with 5 of 6 at zero. Failures are categorical: GPU-generation instability, resource tags, build-date flags, input names, package versions. Once recorded as dead ends they mostly vanish. The only post-transition fix was a novel type-inference bug in graph compilation.
Transcripts show the agent reading the playbook at session start, treating dead ends as do-nots and pipelines as numbered steps. The most autonomous session ran 970 log entries and 110 tool calls with no human, diagnosed a fused-kernel import in a shared operator library, rebuilt package layers, and resubmitted. In another case a lease expiry crashed the implementer; a second server resumed from the shared diff and trajectory, then validated and submitted.
This draws a different boundary than AutoResearch or The AI Scientist: the hard part of industrial auto-research is state, recovery, and institutional memory over multi-day jobs, not the prose of a hypothesis. Playbooks are natural language because, in 31 transcripts, the agent almost never queried numeric metadata. It consumes procedures in the language it reasons in.
The prerequisite is steep. Distributed training, job submission, and metric monitoring already have to exist; the agent is a thin orchestrator. Without that substrate the harness has nowhere to attach. The paper also does not show that the agent found better recommendation architectures. It shows that finishing the run and not repeating known operational failures can be automated.
The authors' future-work list is a list of gaps. There is no proxy model for cheap idea screening, so the idea loop is still gated by full training time. Playbook updates have no validation gate; SkillOpt's ablation says removing that gate hurts, and Auto-RecSys relies on dead ends being correct by construction. The human switch is binary, not confidence-triggered. The design is one researcher over a portfolio of models; shared team queues are unbuilt.
For outside readers the sharper hole is evaluation: no online AUC, NE, or recall table. Several pre-transition "positive" ideas failed on the new baseline and that fact is recorded as a meta-lesson, not as a model-quality result. The reliability curve is from one model. Reading "less human time" as "better recsys metrics" is not supported.