Perplexity launches Q2D-Web, a 190M-doc benchmark and leaderboard for agentic RAG retrieval
perplexity_ai · x · 2026-09-10
Perplexity introduced Q2D-Web (Query2D-Web), a large-scale benchmark and public leaderboard for evaluating embedding models on retrieval in agentic RAG systems.
- Built from 23,000 PII-free production searches collected over nine months, spanning ten languages (65.8% English) and dozens of domains from programming and law to health, science, finance, and travel.
- Agents reformulate user requests into primary and support queries, each evaluated independently with its own relevance judgments.
- The combined set includes 190M web documents, 69,721 agent-reformulated queries, and an average of 99.6 positive judgments per query to reduce false negatives.
- Three relevance sets (agent citations, production web rankings, LLM-expanded judgments) reduce reliance on a single labeling pipeline.
- 13 retrieval models evaluated with Recall@1000: pplx-embed-v1-4b leads Web Ranking (65.73) and Combined (69.11); Nemotron-3-Embed-8B leads Citation (61.68).
- RRF-based subsampling preserves full-corpus rankings using only 31.7% of documents, cutting pplx-embed-v1-4b evaluation from 4,608 to 1,500 H200 GPU-hours.
Related event: Perplexity Releases Q2D-Web, an Open Benchmark for Agentic RAG Retrieval(10 posts)→
More from Infra
- Turn any USB-only printer into an AirPrint printer with a cheap ESP32-S3 — ngxson · 2026-10-27
- Michael Dell: Musk is "the most demanding customer there is" — Colossus servers built in ~6 weeks — elonmusk · 2026-10-27
- AI chip startup Tenstorrent raises $1.5B at $6.2B valuation, more than doubling — Beth_Kindig · 2026-10-27
- Ex-MediaTek expert: Google TPU external revenue $8B this year, $104B by 2028 — RihardJarc · 2026-10-27
- Anthropic discloses up to $120.5B SpaceX compute deal, ~$3.1B monthly at full capacity — XFreeze · 2026-10-27
- Satire: swapping Raft for an LLM-style decision model tanks write throughput 99.98% — irombie · 2026-10-27