AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution
Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles
cs.IR, cs.AI
2026-08-14
Microsoft's AdsWorldEngine trains an ad-serving Orchestrator and its tools in an alternating loop, boosting relevance 80% offline and RPM 22% in a live Copilot A/B test.
Sponsored search can match an ad to a short standalone query. Conversational advertising doesn't get that convenience. A user's commercial intent in a multi-turn chat is often implicit, carried forward by the assistant's own reply, and the system has to decide both whether an ad belongs here at all (too many ads erode trust, too few waste opportunity) and how to turn that implicit intent into something a retrieval system can act on, with no explicit query string to work from. Prior research mostly covered one piece of this pipeline, ad copy generation or insertion timing, not the whole serving loop as one system.
AdsWorldEngine chains four components:
The central contribution is an iterative training loop that alternates between the actor and the tools. The Orchestrator is first trained with supervised fine-tuning plus GRPO using Evaluator rewards. Then, high- and low-reward rollouts from that Orchestrator get converted into preference pairs used to train the tools themselves. Each round, the Orchestrator gets better at using the current tools, and the tools get better at serving an Orchestrator that's rewarded for using them well.
For the subjective calls, like whether an ad belongs in a given turn, the paper introduces label grounded judgment modeling: write an annotation guideline, collect tens of thousands of human labels under it, prompt GPT-5 to generate a reasoning trace conditioned on the guideline and label, filter out traces that contradict the assigned label via a reflection step, then train the judgment model with a cost-sensitive GRPO variant that drops the usual group-standard-deviation reward scaling. That preserves the asymmetric reward gap (false positives penalized harder than false negatives) that standard GRPO's normalization would otherwise wash out.
| Stage | Comparison | Result |
| Opportunity Gate (SFT+cost-sensitive GRPO vs. GPT-5 prompt) | ΔFPR / Balanced Acc | -39.07% / +2.51% |
| Evaluator relevance model (SFT+GRPO vs. GPT-5 prompt) | ΔFPR / Balanced Acc | -12.71% / +7.51% |
| Offline overall (vs. production system) | Diversity / Relevance | +60% / +80% |
| 20-day live A/B test (Microsoft Copilot) | RPM / Ad coverage | +22% / +74% |
One comparison shows why RL, not just SFT, matters here: SFT alone on the Opportunity Gate over-corrects, cutting false-positive rate by 75.42% but dragging true-positive rate down 17.19%, so balanced accuracy actually drops 4.09%. SFT plus cost-sensitive GRPO cuts FPR by 39.07% while holding TPR essentially flat (0.00% change), lifting balanced accuracy 2.51%. SFT alone learns the surface pattern of the reasoning traces; GRPO is what actually moves the decision boundary to where production wants it.
A case study makes the failure mode concrete: a user asks which shade fits them while the assistant walks through undertone-matching steps. Both a GPT-5-prompted judge and the SFT-only AdsWorldEngine mistake this for purchase intent and trigger an ad. Only the fully trained AdsWorldEngine recognizes the assistant is completing the task directly and correctly withholds the ad.
This is one of the few systems that puts "should we show an ad," "which ad," and "how do the tools themselves improve" into a single training loop and actually ships it, running a live A/B test. It demonstrates a broader idea: preference data mined from reinforcement learning on an actor doesn't just improve that actor, it can also be fed back into the tools it calls, letting the whole compound system co-adapt instead of optimizing each module in isolation. Teams building conversational agent systems beyond advertising can lift both the actor-tool co-training loop and the label-grounded-then-filtered judgment modeling recipe directly.
The paper is marked work in progress and doesn't detail model architecture or hyperparameters beyond training set sizes, nor is code released. The Opportunity Gate and Evaluator relevance model are each trained on their own roughly 30,000-example label sets, and inter-annotator agreement on these subjective labels isn't reported, nor is whether the judgment criteria drift as product policy changes. The offline diversity metric relies solely on pairwise cosine similarity among three selected ad embeddings, with no cross-check against actual user satisfaction, and the live A/B test doesn't disclose sample size, confidence intervals, or whether results were broken out by user segment.