Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken, Bo Wang, Denis Bykov
cs.IR, cs.CL
2026-09-08
Q2D-Web pairs 190M web docs with 70k agent queries. pplx-embed-v1-4b hits Recall@1000 69.11; BM25 scores 44.77 but uniquely recovers 14,621 positives.
First-stage retrievers in production agentic RAG scan a web-scale index for queries the agent wrote, not the sentence the user typed. Public IR suites miss that setting on corpus size, query count, and how the query was produced.
MS MARCO Web Search offers 100.9M documents but only 9,374 test queries, each with a single clicked document. BrowseComp-Plus pins evidence to a fixed collection of about 100k documents and 830 queries. TREC Web tracks released about 50 pooled queries a year. Penha et al. already showed that intent-preserving reformulations cut nDCG@10 of retrieval pipelines by about 20% on average. Whatever the first stage drops never reaches the reranker or the citation list.
Q2D-Web is built from nine months of PII-filtered Perplexity production traffic. It keeps 69,721 agent-written queries in ten languages: 12,365 primary rewrites (17.7%) that stay close to the user request, and 57,356 support rewrites (82.3%) that paraphrase, fetch background, or chase neighboring entities. The corpus is the union of a production system's top-5,000 hits per query, then MinHash-LSH dedup at token 5-gram Jaccard 0.975, landing at about 190M pages averaging 13,169 characters. That construction packs hard distractors by design.
Three reusable binary judgment sets sit on the same queries:
Mean positives per query: 99.6. Neural models used for pooling and subcorpus construction were all released before 1 January 2025; the 13 evaluated systems (BM25-tantivy, ten dense encoders from 0.3B to 8B, two late-interaction checkpoints) were released on or after that date.
A full pass of pplx-embed-v1-4b costs 4,608 H200 GPU-hours. Among depth-k pooling, RRF, and uniform random, RRF at k=1000 keeps about 31.7% of the corpus, matches the full ranking with Kendall τb of 1.00, and inflates mean Recall@1000 by 5.1 points. The 4B model then costs about 1,500 GPU-hours. Every system encodes only the first 512 tokens of a document; mean length is about 3.3k tokens.
Primary metric is Recall@1000, because a first stage exists to fill a reranker's candidate pool. Recall@100 and nDCG@10 are reported as well.
Relative order is mostly stable across the three label sets. The winner depends on the metric.
| Model | Comb. R@1000 | Comb. nDCG@10 | Unique positives |
| pplx-embed-v1-4b | 69.11 | 45.84 | 884 |
| Nemotron-3-Embed-8B | 68.58 | 47.44 | 705 |
| EmbeddingGemma-300M | 65.45 | 43.56 | 2,451 |
| Qwen3-Embedding-8B | 64.53 | 42.69 | 596 |
| BM25-tantivy | 44.77 | 30.30 | 14,621 |
Citation Recall@1000 is led by Nemotron-3-Embed-8B at 61.68. Web Ranking and Combined go to pplx-embed-v1-4b at 65.73 and 69.11. Under Combined, Nemotron-8B is stronger at Recall@100 (30.03) and nDCG@10 (47.44): it stacks relevant documents higher, while recovering slightly fewer of them. Qwen3-Embedding scales monotonically from 57.89 at 0.6B to 64.53 at 8B. Among sub-1B dense models, pplx-embed-v1-0.6b leads at 67.02.
Every neural retriever is weaker on support queries than on primary queries. BM25 moves the other way by about 1.9 points, and still finishes last. Its 14,621 unique positives are nearly six times EmbeddingGemma-300M. Late-interaction trails same-size dense models by 3 to 5 Recall@1000 points; the nDCG@10 gap shrinks to at most 1 point.
6,424 labeled positives sit outside every system's top-1000: 51.1% share almost no surface words with the query, 17.7% bury the relevant span deep in a long page, 15.4% cross language. Of 4,000 sampled hard negatives (in every top-1000, labeled irrelevant), 15.8% are actually relevant on re-review. The rest are mostly wrong-aspect (39.0%), partial (17.8%), or related-entity (13.8%).
This is the closest public protocol to production first-stage retrieval for agents. The corpus and queries stay private; only a Hugging Face leaderboard is public, to limit training contamination. For embedding work, the split is practical: Recall@1000 for the first stage, nDCG@10 for shallow ranking, and they crown different models. BM25 still belongs in a fusion mix. Support queries are the actual distribution shift. The RRF subcorpus lets a lab iterate on one 8×H200 node in about eight hours instead of 4,600 GPU-hours per run.
pplx-embed is Perplexity's own family, scored on Perplexity traffic. Keep that on the table.
Labels still miss relevant documents: 15.8% of sampled hard negatives flipped under the same LLM judge. Citation is low-recall by construction, Web Ranking inherits the internal stack, Combined inherits DeepSeek-V4-Flash's tastes. The 512-token cutoff hides most of the body; longer-context spot checks on the subcorpus did not reorder models, and the paper does not claim unbiased full-document scores. Learned sparse retrievers such as SPLADE were skipped for lack of multilingual checkpoints. Whether BM25's unique positives help an ensemble is left untested. The corpus is a union of production top-5,000s, not a random web sample, so difficulty is inflated on purpose and does not transfer to the whole open web.