DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
Xinyu Geng, Xuanhua He, Sixiang Chen, Yanjing Xiao, Fan Zhang, Shijue Huang, Haitao Mi, Zhenwen Liang, Tianqing Fang, Yi R. Fung
cs.CL
2026-07-09
DeepSearch-World is a deterministic, verifiable offline search environment with 420K multi-hop QA tasks. A 9B model self-distills only from its own verified trajectories, with no stronger teacher, reaching 31.2% on BrowseComp and 61.5% on GAIA, up 23.8 points over its Qwen3.5-9B backbone.
Making tool-use agents improve from their own experience has two old paths, each with a knot. SFT distills positive trajectories from the backbone itself but saturates after a few rounds, capped by backbone capability and trajectory diversity. Sparse-reward RL (e.g. GRPO) only rewards at the trajectory end; whether a failure stems from a bad query, wrong tool choice, mis-extracted evidence or a botched answer synthesis, it says nothing. On-policy self-distillation (OPSD) tries to fix sparsity with fine-grained token-level supervision, but in agentic tool use that supervision lives inside each tool call and needs a deterministic environment to keep the teacher distribution stable step by step. DeepSearch-World provides exactly that environment.
DeepSearch-World is a deterministic, verifiable environment built on an offline Wikipedia corpus; its search and browse tools are reproducible, and progress at every tool call can be entity-level verified. It constructs 420K multi-hop QA tasks from entity-level random walks and explicitly supports three cognitive behaviors useful for self-evolution: progress verification, grounded reflection and failure recovery.
On top sits the DeepSearch-Evolve self-distillation framework. The key is scaffolded process supervision: during rollout the teacher explicitly tracks progress, evidence, failed attempts and recovery, writing these process signals into the trajectory. The scaffold is used only at trajectory generation, never exposed to the student; a scaffold-to-ReAct conversion distills the planning, memory and error-correction it induced into standard ReAct format. Each round generates 10,000 trajectories, and 4,000 that pass rejection sampling and quality filtering trigger one training step, with trajectory length capped at 30 steps. The loop runs 11 rounds from a Qwen3.5-9B backbone, followed by a GRPO pass on 1,600 real-tool instances to close the offline-to-real gap.
Across seven deep-search and reasoning benchmarks, DeepSearch-World-9B uses no stronger-teacher distillation, learning only from its own environment-verified rollouts:
| Benchmark | Score |
| BrowseComp | 31.2% |
| GAIA | 61.5% |
| HotpotQA | 93.4% |
Against the Qwen3.5-9B-Instruct backbone under matched tools, it rises across the board: +23.8 on BrowseComp, +37.6 on GAIA, +48.1 on HotpotQA. Behavior diverges sharply: the backbone quits after 4.7 rounds on average with premature answers, while DeepSearch-World sustains 18.0 rounds, lifts visit calls from 0.9 to 5.4, and raises the advanced-capability score from 19% to 70%. In ablations, rejection sampling is the main driver (SearchQA 46.4 to 54.9), reaching 58.2 with quality filtering; removing reflection rewriting collapses it to 16.7, since raw reflections carry artifacts like [REFLECTION] tokens that pollute the think distribution. A 420K-sample pool reaches a higher validation plateau than 100K, showing self-evolution rides on data-pool diversity, not repeated exposure.
For teams working on agent self-evolution, the transferable claim is that a verifiable, deterministic environment can substitute for part of the supervision traditionally supplied by a stronger teacher or external synthetic pipelines. It moves "agents learning from their own experience" from "SFT saturates in a few rounds" to a regime that iterates 11 rounds stably and generalizes across benchmarks. The separation of scaffold-at-generation and ReAct-at-training is directly reusable for other long-horizon tool tasks.
The environment is built on Wikipedia, so coverage and domain diversity are limited; behavior on broader knowledge sources is unverified. The update rule is iterative SFT; the authors admit RL- or OPSD-style updates may be stronger, but how to inject planning, error recovery and tool strategy into RL training remains open. The low BrowseComp-ZH score is expected, since training uses only English trajectories, so cross-lingual transfer is partial. The scaffold teacher still needs a reasonably strong backbone to produce high-quality trajectories, so "no stronger-teacher distillation" strictly means "no direct use of a stronger model's trajectories," while a backbone capable of good trajectories is still a prerequisite.