DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, Pang Wei Koh
ICML 2026
cs.CL, cs.AI, cs.LG
2025-11-25
RLER trains Qwen3-8B into DR Tulu. It averages 65.6 on four long-form deep-research benchmarks, 15.6 above Tongyi DR, at about 1/1000th the per-query cost of OpenAI DR.
Most open deep-research agents are trained with reinforcement learning on short, verifiable QA. A gold answer exists, so the reward is cheap. Real research work is not like that. The user wants a long, sourced report, the quality bar is underspecified, and checking facts needs external knowledge the policy does not hold.
Static rubrics miss evidence the model later retrieves. Closed-book LM-written rubrics miss the same evidence and are easy to game. That is why systems such as Tongyi DeepResearch look strong on short QA and drop once the task becomes long-form.
RLER keeps a per-prompt rubric buffer that changes with the policy. Each prompt starts with persistent, search-grounded criteria. During training the buffer also holds rotating items written from the current rollouts.
At each step the policy draws 8 tool-using rollouts on the same prompt. The rubric model sees two kinds of information the policy did not have while writing: documents gathered across those search traces, and contrast among the answers. It then writes positive items (new, relevant points the old buffer missed) and negative items (shared reward hacks, such as copying retrieved text to inflate citation precision).
The buffer is capped. Zero-variance items are dropped; the rest are ranked by the standard deviation of rollout scores and only the most discriminative Kmax items are kept. Format, search, and citation auxiliaries sit on top. Citation uses a claim-level F1: extract claims, then ask a judge whether the cited snippets support them and whether they are relevant.
Training starts from Qwen3-8B. GPT-5 produces about 16K auto-search trajectories for SFT (8 H100s, 136 GPU-hours). RL is GRPO on roughly 9K long-form prompts with asynchronous tool calls. The tool set is googlesearch, webbrowse, and papersearch, capped at 10 calls per query at inference. The main run uses GPT-4.1 to write rubrics and GPT-4.1-mini to score them.
On four long-form benchmarks, DR Tulu-8B averages 65.6.
| System | SQAv2 | HealthBench | ResearchQA | DRB | Avg. |
| Tongyi DR-30B | 46.5 | 46.2 | 66.7 | 40.6 | 50.0 |
| OpenAI DR | 79.6 | 53.8 | 79.2 | 46.9 | 64.9 |
| GPT-5 + Search | 74.8 | 59.5 | 78.2 | 50.7 | 65.8 |
| DR Tulu-8B (SFT) | 72.3 | 38.1 | 68.5 | 39.0 | 53.9 |
| DR Tulu-8B (RL) | 88.3 | 52.8 | 75.7 | 45.4 | 65.6 |
RL adds 6.4 to 16.0 points on top of SFT. The 16.0-point jump on SQAv2 is the largest, because that benchmark scores citations and prior open agents mostly emit none. On ScholarQA-CSv2 a query costs about $0.0019 against $1.80 for OpenAI DR, roughly 1000x.
Short-form QA is not washed out. The SimpleQA / 2Wiki / WebWalker average moves from 58.0 after SFT to 61.3 after RL. Tool mix follows the task: papersearch is about 90% of calls on SQAv2; web search plus browsing is about 55% on DeepResearchBench.
Dropping evolving items and keeping only the initial search rubrics costs up to about 2 average points, and the gap widens later in training. Swapping GPT-4.1 for Qwen3-8B as both rubric writer and judge still yields +4.4 over SFT after 1000 steps, 1.3 behind the GPT configuration. The recipe does not depend on distilling a stronger closed judge.
This is a fully open long-form deep-research stack: data, code, tools, and training. An 8B model can beat larger open agents and some closed pipelines on scientific synthesis, and it picks tools instead of hard-coding one search source.
It is a training recipe, not a new architecture. 65.6 is 0.7 above OpenAI DR and 0.2 below GPT-5 + Search. Closed systems still lead on HealthBench and ResearchQA. A large share of the open-model gap is "does it emit checkable citations," not deeper reasoning.
The paper has no dedicated limitations section. The analysis notes that training reward and downstream scores can diverge: a higher train reward need not mean a higher benchmark. Most closed numbers are reused from leaderboards; expensive systems were run on 100-example subsets. The SQAv2 lead is inflated by rivals scoring zero on citations. The final RL run used about 27,000 GPU-hours.
The Impact Statement flags the usual research-assistant failure modes: fluent falsehoods, selective citation, biased synthesis. GeneticDiseasesQA has 47 expert questions; open agents without traceable citations are simply dropped, so the comparison is thin. Evolving rubrics add only about 2 points over static search-based ones. The main lift is "ground the rubric in search," not "it must evolve online."