Deep Search in Real-World Web Environments
Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, Jiwei Fang, Yu Yan, Yini Huang, Chen Chen, Ruoxi Cheng, Yuzhi He, Yuhui Zhang, Xiaoxing Hu, Chen Liu, Jianbo Lin, Yijiang Li, Xiaobin Hu, Menglong Li, Zhuoran Lu, Weishi Wang, Zhongqi Yue, Chengwei Qin
cs.IR, cs.MM, cs.SE
2026-10-20
On 200 live-web tasks that chain a video anchor across formats and interfaces, GPT-6 Astra scores 39.0%; 1,000 human attempts score 21.1%.
Deep Search benchmarks rarely force an agent to finish an evidence chain on the live web. A real page mixes text, images, audio, and video, stores that evidence as HTML, PDF, or Parquet, and often reveals it only after a link hop, a parameter change, a time seek, or a download. Drop one of those layers and the answer breaks.
OpenDS is multi-step factual QA on the open web. The input is a public video and a question. Source URLs are withheld. The model has to name the anchor in the video, then join evidence from several independent sites into one checkable answer.
Items grow out of video. Annotators set the anchor from 500 candidates. The entity must be recognizable from picture and sound, and it must map to exactly one Wikidata entry, so the start cannot collapse into text search on the video page. They then expand hop by hop with Google Search. Every hop needs support from more than one independent page, and the chain stops at a unique terminal node. The question is that chain written backwards. Intermediate entities survive only as indirect references. Ten reviewers check that the evidence is reachable and that the answer is unique.
Two hundred tasks remain, across eight domains. The chains hold 787 entries, 3.94 per task, covering 4 modalities, 42 formats, and 7 access types. Text appears in 99.5% of tasks. Images, video, and audio appear in 44.5%, 35.5%, and 16.0%. Link following, download-then-read, temporal indexing, and parameter changes cover 67.0%, 60.0%, 47.5%, and 42.0%.
Twelve models share search, a browser GUI, and frame and subtitle tools. Each task gets at most one hour and 100 calls. Past either cap, tools switch off and the model answers from what it already has. Qwen3.8-27B judges semantic equivalence. Fifty people make 1,000 attempts with a normal browser and no LLM. Low, Medium, and High follow how many of five solvers are correct: 54, 61, and 85 tasks.
| Method | Metric | Result |
| GPT-6 Astra | Accuracy | 39.0% (78/200); Low/Med/High 59.26%/40.98%/24.71% |
| Claude Opus 5.5 | Accuracy | 38.0% (76/200) |
| Gemini 3.8 Flash | Accuracy | 36.5%; best on High at 28.24% |
| Humans, 1,000 attempts | Accuracy | 21.1%; 85 tasks missed by all five |
| Qwen3.8-Flash | Budget | 80.0% hit the call cap; natural stop 6.5% |
Equal-weight accuracy falls from 39.66% on Low to 23.50% on Medium and 16.76% on High. Astra still misses 122 tasks. It and GPT-5.6 Sol use almost the same tokens, about 187M versus 186M, and sit 11 points apart. Gemini uses about 5.22 times Astra's tokens. DeepSeek-V4-Pro uses about 2.39 times Astra's wall-clock time and scores 12.5%. Opus 5.5 and Qwen3.8-Flash put 51.52% and 49.42% of calls into the GUI, with accuracies of 38.0% and 12.0%. Failed calls are 2.83% to 9.42% of calls but 19.27% to 61.90% of tool time. Humans average 12.92 hours on each set of 20 questions, about 646 hours in total, and a majority solves only 29 tasks.
Under one budget, more tool use is a weak signal. Qwen3.8-Flash averages 89.94 calls and scores 12.0%. Astra averages 51.05 calls and 18.66 minutes, hits a limit on 25.5% of tasks, stops on its own on 69.0%, and still scores 39.0%. On the 85 High tasks, Gemini's 28.24% beats Astra's 24.71%. The questions hide intermediate entities, so they are harsher than a routine lookup. They separate wasted search from a wrong stop. They are a poor acceptance test for a product.
The videos are archived with the tasks. The evidence pages are not, and link survival is not reported, so 39.0% and 21.1% will be hard to repeat. Humans could not use an LLM and had no shared time cap. Models had search and a GUI, capped at one hour and 100 calls. Beating the human number only shows an edge over solvers who were barred from models. All five humans missed 42.5% of tasks, while the annotation path still exists, so inverted wording supplies part of the difficulty. The judge is in the same Qwen family as two contestants. Agreement with human grading is unreported, and the main numbers look like one run per task.