Perplexity releases Q2D-Web: a 190M-document retrieval benchmark for agentic RAG
antoine_chaffin · x · 2026-09-10
Perplexity and collaborators released Q2D-Web, a large-scale retrieval benchmark and public leaderboard for agentic RAG systems (arXiv preprint).
- 190M-document web corpus paired with 70K agent-reformulated queries in 10 languages, derived from real production user queries.
- Motivation: existing benchmarks either have huge corpora but few queries, or many queries but only million-scale corpora — and none test machine-reformulated queries, whose distribution differs from human search.
- Three relevance judgment sets: agent citations, production rankings, and a union plus LLM-based judgments to reduce false negatives.
- Benchmarked 13 retrievers (lexical, dense, late-interaction); relative ordering is largely insensitive to judgment-set choice.
- Public leaderboard included.
Related event: Perplexity Releases Q2D-Web, an Open Benchmark for Agentic RAG Retrieval(10 posts)→
More from Research
- AutoResearch agent produces weird 'forking' loss curves, and the team dug into the science — ZimingLiu11 · 2026-10-27
- No π0 weights changed: Motoniq layer lifts real-robot precision fitting from 0/10 to 9/10 — hbouammar · 2026-10-27
- JockeyBench uses Japanese horse racing to test LLM agents — all betting strategies lose money — Zhongwei Xie · 2026-10-27
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27