Agentic ML Exploration (A-MLE) for Ads Ranking
Erwin Gao, Vinodh Kumar Sunkara, Jingyi Guan, Qinjin Jia, Hangjun Xu, Xiang Ji, Sherman Wong, Surya Teja Chavali, Pratik Vaishnavi, Aryan Pandhi, Xiaoyu Deng, Zhaodong Wang, Samarth Inani, Fan Yang, Jakob Moberg, Zoe Zu, Nicolas Bievre, Sami Khenissi, Amit Jaspal, Ehsan Fakharizadi, Srinidhi Viswanathan, Dorothy Sun, Abishek Vanam, Sneha Iyer, Sheela Yadawad, Wenjie Chen, Gaby Nahum, Junhua Gu, Peter Chu, Yucheng Liu, Xin Zhao, Vitor Cid, Chaorong Chen, Vijay Pappu, Ashwin Kumar, Wenlin Chen, Ben Schulte, Deepak Chandra, Ritwik Tewari
cs.AI
2026-09-08
A-MLE has an LLM agent run hypothesis-to-eval across ads ranking models. Domain skills hit 68% L1 accuracy; multi-source search cuts relative error 2.56% on model M*.
An industrial ads ranking stack is a portfolio of dozens of models. Objectives (click, conversion, view), surfaces, and architecture families (deep cross network, deep interest network, multi-tower) diverge, and so do data and infrastructure constraints. Porting a proven trick from one model to another is expensive. The bottleneck is rarely model capacity or training compute. It is human iteration: hypothesis, code, train, debug, evaluate, write a launch proposal. A senior engineer spends days to weeks on one model. With a finite staff, only a thin slice of model-by-technique pairs ever get tried. Long-tail models sit idle even when a neighbor has already landed the same class of change.
A-MLE splits one exploration session into five stages, run by a single tool-using LLM agent. Humans review at stage boundaries rather than at every keystroke.
Hypothesis generation reads the live training config, rolling baseline, and attempt history, then scores a small candidate set from model-internal-state analyzers, training-efficiency analyzers, and recent-literature retrievers. Strategy planning interleaves isolated tests with combinations of the most promising candidates, under a budget of runs, compute, or wall time. Experiment execution edits a sandboxed copy, type-checks and unit-tests, smoke-runs, then submits full training. The agent separates infrastructure failures from genuine divergence and retries, patches, or reroutes remaining compute. Result analysis tests against a rolling baseline, not a frozen snapshot, decomposes by ad and user segment, and auto-reruns when within-run variance is too high. The shared substrate is a long-lived markdown tree in source control, keyed by technique and by model, so a win on one architecture becomes eligible on a similar one.
The skill library covers codebase navigation, training config edits, launch and monitoring, offline evaluation, and proposal writing. Each session is parameterized by a (model, objective, compute) triple and ends with a documented proposal or a documented null result.
Throughput, training success, and proposal acceptance are described as several times the manual baseline, or much higher, without exact multipliers. The numbered results sit elsewhere.
On the L1 tool-availability bench for experimental model M, the domain-equipped agent scores 68% overall, versus 16% for a generic ML agent with portfolio tools but no domain skills, and 8% for a bare LLM. The widest gap is job-config modification: the domain agent solves every question; the generic setups stay below 40%. At L2, the domain agent completes four multi-step tasks end to end: baseline refresh, variance test, config-change comparison, and batch offline eval. What separates it from a single-turn loop is a waiting operator for multi-hour jobs, infra-versus-divergence triage, and summaries that survive noisy multi-run logs.
At L3, A-MLE posted measurable offline gains on a majority of evaluated models. Headlines on M, a lighter regression-objective ranking model:
| Configuration | Relative error drop | Train QPS |
| Single-hypothesis arch scale-up | +0.44% | Neutral |
| Multi-round arch exploration | +0.58% | Neutral |
| Multi-source (arch + efficiency) | +2.56% | +0.42% |
Holding the agent loop, skills, and prompts fixed and swapping the LLM, Sonnet 3.5+, Gemini 2.5, and GPT-5 land in the high-90s on L2 task completeness. Weaker models hallucinate workflow IDs or fail to wait. At L3, Gemini 2.5 and GPT-5 explore more aggressively under a basic prompt; the Sonnet family is conservative. Under a stressful competitive prompt the ranking flips: Sonnet 4.0 posts the largest improvement of any configuration (about 2.6×10⁻² rMSE), while GPT-5 turns more conservative and gives back most of its basic-prompt gain.
Across the portfolio the strongest pattern is technique transfer: an architecture edit or embedding-feature variant that already passed gating on one model gets tried on a structurally similar model that had never run it. Self-supervised pretraining is attempted on many models and usually passes; generic optimizer and loss tweaks are attempted widely with mixed gating; architecture scaling covers few models. Models that just took a non-trivial baseline refresh are a weak spot, because hypotheses stay calibrated to the previous version.
For ads and recommender teams, this moves AutoML off a hyperparameter grid and onto code-level edits of a mature baseline. The objective is composite: offline metric gain, infrastructure feasibility, statistical significance, and proposal quality, not a Kaggle score. The main payoff is breadth on the long tail, where senior attention is scarce and nearby models already hold validated tricks.
The design lesson is blunt. Reliability tracks the orchestration harness (skill coverage, rolling-baseline stats, retries against infra noise) more than raw model reasoning. Stronger LLMs will close the hypothesis-quality gap faster than the harness gap. Humans still gate each stage, which caps the blast radius of a hallucinated code edit or a misread eval.
Throughput, success rate, and acceptance rate stay qualitative: no engineer-week counts, no exact multiplier versus the semi-automated baseline. Models are anonymized as M1, M2, M. Portfolio size, evaluation-window length, and online A/B are not reported. Technique-family counts are bucketed as many / several / few, with no exact pass rates. The +2.56% L3 headline is on a lighter experimental model and should not be read as the lift on the largest conversion models. Failure modes are documented clearly: hallucinated APIs, baseline drift wiping an offline win, treating a transient incident as divergence, promoting on a single seed. Stressful prompts push model families in opposite directions, so "make the prompt more competitive" is not a monotone upgrade.