Training Documents Reranker with Search Rubrics for Deep Research Agent
Wenhan Liu, Yu Lu, Qiaolin Xia, Hui Xu, Tong Zhao, Jian Xi, Yutao Zhu, Haijin Liang, Haibo Shi, Hao Wang, Zhicheng Dou
cs.IR, cs.AI, cs.CL
2026-08-04
Deep-research agents retrieve top-k documents that are each relevant, yet as a set may miss diversity, repeat each other, or include weak sources. Tencent proposes query-specific search rubrics (relevance/conciseness/consistency plus authority/timeliness) and trains RubricRanker (Qwen3-8B) with SFT plus RL, beating the strongest baseline by 2.6 across four deep-research benchmarks while cutting the agent's search calls.
Deep research agents answer complex questions by repeatedly retrieving, reasoning, and retrieving again. At each step they issue a sub-query to a search engine and get back a batch of web documents. Existing retrievers and rerankers score by single-document relevance to pick the top-k, but documents that are each individually relevant may not form a set that satisfies the agent's complex needs: they may miss an important angle, repeat each other and waste context budget, or include a non-authoritative source that leads the agent astray.
The paper's example is concrete. A user asks whether they might be depressed and what to do; the agent issues the sub-query "how to treat depression." The three retrieved documents are all relevant — one on medication, two on psychotherapy. But the set has three problems: it omits self-regulation as a treatment type; the two psychotherapy docs are redundant; and the third comes from a non-authoritative personal blog that could yield unreliable medical advice. Single-document relevance cannot fix set quality.
RubricRanker's idea: write what a high-quality document set should satisfy as explicit query-specific rubrics (scoring criteria), then train a reranker with them so it selects a good subset directly, with no rubric input needed at inference.
Rubric construction has three layers. First, fixed "meta rubrics" at two levels: set-level (relevance, conciseness, consistency) and document-level (source authority, timeliness). Then GPT-5.1 (with web search) synthesizes a high-quality reference answer per query as a blueprint, from which it expands the meta rubrics into query-specific rubrics, each weighted 1 to 5. Training queries come from deep-research datasets (OpenScholar, SearchArena, etc., run through the Dr-Tulu-8B agent to get sub-queries) and RAG datasets (HotpotQA, NQ).
Training is two-stage. Stage one, rubrics-guided SFT: GPT-5.1 acts as a teacher, selecting the best subset under rubric guidance as silver labels to cold-start the reranker. Stage two, rubric-based RL with GRPO: the reward is a weight-aggregated combination of set-level and document-level rubric scores (the GPT-5.1 judge scores the whole set for set-level rubrics, scores each selected doc and averages for document-level rubrics), plus a format reward. The backbone is Qwen3-8B; at inference it takes the query and candidate list and outputs selected document IDs directly.
Two scenarios: deep research (Dr-Tulu-8B agent, Google Search API retrieval, rerank top-30 and keep top-5) and RAG (Qwen3-8B generation, BGE retrieval over Wikipedia, exact match).
| Scenario | Best baseline | RubricRanker | Gain |
| Deep research, 4-benchmark avg | Rank4Gen 57.5 | 60.1 | +2.6 |
| RAG, 5-benchmark avg EM | Rank4Gen 38.2 | 40.0 | +1.8 |
On deep research, WebWalkerQA 58.0, HealthBench 61.5, and ResearchQA 74.2 lead clearly, while DRB at 46.8 is close to the baseline. On RAG, all five (HotpotQA 38.0, PopQA 42.2, etc.) are best. Two observations stand out. First, generation-oriented rerankers (Rank4Gen, SetR) help on RAG but their gain nearly vanishes on deep research, so models trained on closed-form RAG do not transfer to open-ended deep research, which is exactly where rubric training earns its keep. Second, any reranking beats raw retrieval (deep research 54.0 to 56.1–60.1; RAG 34.3 to 35.3–40.0).
Ablations: removing RL drops the score from 52.5 to 51.1; removing cold-start SFT drops it to 48.3 (SFT matters more for stable RL initialization); dropping rubrics from the labels drops to 49.2; using plain relevance-ranking labels drops to 48.1. It also cuts search calls: on HealthBench the agent's average searches fall from 3.2 (RankT5) and 3.4 (Rank4Gen) to 2.9 (relative drops of 9.4% and 14.7%), because better evidence per step lets the agent stop sooner.
This shifts the reranking objective from single-document relevance to document-set quality, a lens relevant to anyone building RAG or agent retrieval. Using rubrics as supervision is interpretable: you can see which criteria constrain each query and with what weight, which helps debugging. For production systems, reducing the agent's search calls means lower latency and cost, a practical win.
It is an 8B model that needs neither rubrics nor GPT-5.1 at inference, so deployment is cheap.
The authors note that rubric rewards rely on GPT-5.1, which is costly at training scale (they plan a specialized reward model). Because LLM evaluation of deep research is expensive and slow, only a subset was run (100–200 queries per benchmark), not the full sets. Final evaluation still depends on agent generation quality: even with good documents selected, the agent can still err in reasoning or generation, and there is no objective metric for document-set quality itself.
One more concern: evaluation uses an LLM judge scoring by rubrics, while rubrics are also this method's core, creating a mild circularity of using the same kind of tool as both referee and training signal. Treat absolute numbers with caution; the relative ranking and the drop in search calls are the more robust signals.