DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, Sasha Goldshtein, Dipanjan Das
www
cs.CL
2026-01-29
Google's 900-prompt DeepSearchQA scores exhaustive search as set matching. Gemini Deep Research hits 66.09% fully correct and 81.90 F1; Flash fully-incorrect jumps to 45.27%.
Benchmarks like SimpleQA, BrowseComp, GAIA, and Humanity's Last Exam mostly ask for one right answer. That format is cheap to grade. It also trains search agents to hunt a needle: high precision, stop on first hit.
A lot of real research wants a roster. Users need every entity that survives a stack of filters: which US states sat in a given CDC death-rate band for the leading cause of death in 2019-2020; which English cities clear successive cuts on house prices, green space, employment, and clean-air zones. No single page holds the full list. The paper names the missing eval the Comprehensiveness Gap.
DeepSearchQA is 900 expert-written prompts across 17 fields, spanning politics, finance, science, health, history, geography, and media. Each item is a causal chain, so step n consumes the set from step n-1 and stresses long-horizon planning. Prompts are time-anchored or tied to static sources, such as "according to the 2020 Census", to slow gold-label drift on the live web.
Answers come in two types. Single Answer is still one entity, but the fact is obscure and sources conflict. Set Answer is an enumeration or a composite of sub-questions; the agent must submit a complete set. Quality control is three-phase: three reviewers research independently without seeing the gold, then compare, then resolve conflicts. Ambiguous prompts are dropped.
Tasks are tagged by cognitive load:
Scoring is outcome-only. Trajectories do not count. The primary metric is set-level precision, recall, and F1. Responses also fall into four buckets: Fully Correct (semantic set equality), Fully Incorrect (empty intersection), Partially Correct, and Correct with Extraneous (perfect recall plus extra junk, the hedging mode). Gemini 2.5 Flash judges semantic match in a zero-shot setup. Kaggle grades independently and hosts a public leaderboard.
The design stresses three skills current boards barely isolate: collating fragments across many sources, resolving duplicate entities, and deciding when to stop in an open search space. The hard part of stopping is telling "I have not found it yet" from "it does not exist."
Kaggle's main table (Fully Correct / Fully Incorrect / extraneous extras / F1):
| Model | Fully Correct | Fully Incorrect | Extraneous | F1 |
| Gemini Deep Research Agent | 66.09 | 9.95 | 10.30 | 81.90 |
| GPT-5 Pro High Reasoning | 65.18 | 14.13 | 8.12 | 78.98 |
| GPT-5 High Reasoning | 59.41 | 19.91 | 6.56 | 73.24 |
| Gemini 3 Pro Preview | 56.56 | 12.78 | 9.89 | 76.86 |
| o3 Deep Research | 44.24 | 20.09 | 11.74 | 66.45 |
| o4 Mini Deep Research | 40.36 | 24.19 | 7.80 | 61.76 |
| Gemini 2.5 Flash | 25.92 | 45.27 | 5.90 | 42.99 |
| Claude 4.5 Opus | 24.01 | 50.66 | 4.18 | 40.20 |
| Claude 4.5 Sonnet | 16.04 | 64.25 | 2.90 | 27.85 |
| Claude 4.5 Haiku | 12.78 | 71.00 | 1.89 | 22.24 |
Gemini Deep Research and GPT-5 Pro are tied on fully-correct within 95% CIs (66.09 vs 65.18), but Gemini's fully-incorrect rate is lower: 9.95% vs 14.13%. The paper reads that as fewer total trajectory collapses. Claude 4.5 models appear as standalone reasoners, not Deep Research loops, so the cliff on that side of the table is not a fair model bake-off.
Sampling Gemini Deep Research n times lifts fully-correct from 67.18% (n=1) to 85.71% (n=8); n=2 is 74.51%, n=4 is 81.72%. Cheap models fall off a step, not a slope: Gemini 2.5 Flash F1 is 42.99 with a 45.27% fully-incorrect rate, about five times the leader. Routing research jobs to Flash or Haiku is not a small quality tax. Search paths go to zero.
F1 81.90 versus fully-correct 66.09 is a 15-point Last Mile Problem (about 13 points for GPT-5 Pro). Two opposite errors sit in that gap: under-retrieval that misses long-tail entities, and over-retrieval that cannot stop and pads adjacent topics.
Three failure sketches. Gemini ranks EU freight countries with estimated tonnage instead of Eurostat figures and gets the order wrong. GPT-5 Pro finds an Excel file of US airport on-time ranks, says it cannot open spreadsheets in chat, and quits; Gemini switches sources and is partially right. GPT-5 Pro lists botanists in Oxford's Sherardian Library and then skips the filter "never held the Sherardian chair."
If a product ships on BrowseComp or HLE scores alone, it will undercount how hard exhaustive listing is. DeepSearchQA turns the explore-versus-verify trade-off into set matching. High F1 with middling fully-correct means "mostly found" is not the same as "a clean list you can file."
Test-time sampling moves fully-correct from about 67% to about 86% on this set, so extra compute does buy completeness. Cutting the model class in the other direction spikes fully-incorrect into the 45-71% band.
For products that must return a complete roster, watch fully-correct and the extraneous rate, not F1 alone. Stopping policy, spreadsheet tool use, and entity resolution currently look more valuable than one more search hop.
The authors flag two limits. Outcome-only scoring cannot tell a reasoned list from a lucky one. Live pages change or vanish, so static gold will rot and needs periodic human refresh.
A few more gaps sit in the writeup. The judge is Gemini 2.5 Flash while several Gemini systems sit on the same board; there is no independent-judge ablation. Claude lacks a matched Deep Research scaffold, so the table is not a model ranking. The claimed cost edge cites public price pages, not dollars spent on this benchmark. The main-table 66.09% and the n=1 sampling figure 67.18% do not match, and the paper never says whether those are the same eval run.