Perplexity Releases Q2D-Web, First Open Benchmark for Agentic RAG Retrieval

Perplexity released and open-sourced Q2D-Web (Query2Doc-Web) on September 9–10 — the first public benchmark and leaderboard for first-stage retrieval in agentic RAG. It measures how well embedding models handle agent-rewritten queries in large-scale web retrieval, filling a gap left by existing public benchmarks for this scenario.

Confirmed

• The corpus contains roughly 190 million web documents, paired with about 70,000 search requests derived from real user queries rewritten by agents, spanning ten languages.
• The data foundation includes 23,000 PII-scrubbed production queries collected over nine months.
• The evaluation metric is Recall@1000 rather than the common nDCG@10, better suited to large-scale web retrieval.
• On methodology (per @perplexityai's follow-up post), the benchmark combines three relevance sets — agent citations, production web rankings, and LLM judgments — to reduce false negatives and reliance on a single annotation pipeline, and to examine how the definition of relevance itself affects model performance.
• Both the benchmark and leaderboard are publicly available.

Why it matters

As agents increasingly rewrite user queries before retrieval, traditional benchmarks built on short human-written queries no longer reflect real workloads. Grounded in production data and agent-rewritten queries, Q2D-Web offers a more realistic evaluation standard for embedding models in agentic RAG, and its multiple relevance annotation schemes also offer useful lessons for benchmark design methodology itself.

2026-09-09 ~ 2026-09-10 · 8 related posts

• Perplexity Releases Q2D-Web: 190M-Doc Benchmark for Agentic RAG Retrieval — _reachsumit · 2026-09-09
• [source] Perplexity launches Q2D-Web, a benchmark and leaderboard for agentic RAG retrieval — perplexity_ai · 2026-09-10
• [source] Q2D-Web methodology: three relevance label sets and hard distractors cut false negatives — perplexity_ai · 2026-09-10
• Perplexity's Q2D-Web: 190M Documents, 70k Agent-Rewritten Queries for Retrieval Eval — CShorten30 · 2026-09-10
• Perplexity launches Q2D-Web benchmark and leaderboard for agentic web search retrieval — andrewgwils · 2026-09-10
• Q2D-Web: new retrieval benchmark with 190M web docs and 70k LLM-rewritten queries — antoine_chaffin · 2026-09-10

2 near-duplicate retellings: perplexity_ai · antoine_chaffin