Perplexity Releases Q2D-Web, an Open Benchmark for Agentic RAG Retrieval

Perplexity released and open-sourced Q2D-Web (Query2Doc-Web) on September 9-10, the first public benchmark and leaderboard for first-stage retrieval in agentic RAG. It measures how embedding models handle agent-rewritten queries over large-scale web retrieval, filling a gap left by existing public benchmarks. The paper is on arXiv, with the benchmark and leaderboard publicly available; collaborators include AI researcher Deniz Yarats and others.

Confirmed

• The corpus contains 190M web documents with 70K search requests derived from real user queries rewritten by agents, covering ten languages.
• The data foundation includes 23,000 production queries collected over nine months with PII removed.
• The metric is Recall@1000 rather than the common nDCG@10, better suited to large-scale web retrieval.
• Per a follow-up post by @perplexityai, the benchmark combines three relevance sets—agent citations, production web rankings, and LLM judgments—to reduce false negatives and dependence on a single annotation pipeline, and examines how relevance definitions affect model performance.

Why it matters

As agents increasingly rewrite user queries before retrieval, traditional short-query benchmarks no longer reflect real workloads. Built on production data and agent-rewritten queries, Q2D-Web offers a more realistic evaluation standard for embedding models in agentic RAG, and its multi-pipeline relevance annotation approach is itself instructive for benchmark design methodology.

2026-09-09 ~ 2026-09-10 · 10 related posts

• Perplexity Releases Q2D-Web: 190M-Doc Benchmark for Agentic RAG Retrieval — _reachsumit · 2026-09-09
• [source] Perplexity launches Q2D-Web, a benchmark and leaderboard for agentic RAG retrieval — perplexity_ai · 2026-09-10
• [source] Q2D-Web methodology: three relevance label sets and hard distractors cut false negatives — perplexity_ai · 2026-09-10
• Perplexity's Q2D-Web: 190M Documents, 70k Agent-Rewritten Queries for Retrieval Eval — CShorten30 · 2026-09-10
• Perplexity launches Q2D-Web benchmark and leaderboard for agentic web search retrieval — andrewgwils · 2026-09-10
• Q2D-Web: new retrieval benchmark with 190M web docs and 70k LLM-rewritten queries — antoine_chaffin · 2026-09-10

4 near-duplicate retellings: perplexity_ai · antoine_chaffin · antoine_chaffin · antoine_chaffin