Q2D-Web methodology: three relevance label sets and hard distractors cut false negatives
perplexity_ai · x · 2026-09-10
Perplexity shared methodology details for Q2D-Web: the benchmark combines three relevance sets — agent citations, production web rankings, and expansions via LLM judgments — to reduce false negatives and reliance on a single labeling pipeline, and to test how relevance definitions affect model performance.
The corpus takes the top 5,000 production retrieval results per query, deduplicated with MinHash-LSH. Every document plausibly matches at least one query, including difficult distractors that match the topic but miss a required date, entity, or version.
Related event: Perplexity Releases Q2D-Web, an Open Benchmark for Agentic RAG Retrieval(10 posts)→
More from Research
- AutoResearch agent produces weird 'forking' loss curves, and the team dug into the science — ZimingLiu11 · 2026-10-27
- No π0 weights changed: Motoniq layer lifts real-robot precision fitting from 0/10 to 9/10 — hbouammar · 2026-10-27
- MICE paper pushes cross-encoders from re-ranking to end-to-end retrieval — antoine_chaffin · 2026-10-27
- JockeyBench uses Japanese horse racing to test LLM agents — all betting strategies lose money — Zhongwei Xie · 2026-10-27
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27