Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search
Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang, Krishna Kamath, Xi Chen, Roberto Konow, Kurchi Subhra Hazra
cs.IR, cs.LG
2026-08-04
Pinterest fine-tunes Qwen3-VL to label search relevance 5-point, matching humans 82.9% and cutting A/B minimum detectable effect sixfold at 99.98% lower labeling cost.
Personalized search has to watch two metric families at once: user engagement (clicks, saves) and semantic relevance. Engagement is polluted by position, presentation, and attention effects, so a personalization change can lift engagement while sneaking irrelevant content into the top slots. Relevance is the guardrail alongside engagement. The problem is that relevance has always relied on human annotation, which is expensive, slow, and hard to scale, and that pushes the minimum detectable effect (MDE, the smallest metric change an experiment can reliably see) up, so only large moves are visible and small or segment-level effects are missed.
Pinterest fine-tunes the open-source Qwen3-VL (settling on the 4B variant) as a relevance rater. The input is the Pin image plus a full set of text metadata (title, description, landing-page title and description, board titles it sits in, historically high-engagement queries), and the output is a 1-to-5 relevance grade. Training uses about 0.8M human-annotated query-Pin pairs, with 20K held out for model selection.
What actually decides experiment sensitivity is the sampling design, not the model. Once VLM labeling cuts cost and turnaround, the authors move from simple random sampling to stratified sampling: strata are query interest category crossed with popularity (head, torso, tail, single), with optimal allocation. Between-stratum variance is the dominant source of relevance variance, and stratification removes it. The metric is sDCG@K (an nDCG variant assuming infinite L5 documents) with K=25, and heterogeneous treatment effects are computed by popularity and category with Benjamini-Hochberg FDR control.
| Metric | Value |
| VLM-human exact agreement | 82.9% (94.2% within one level) |
| Quadratic weighted Kappa | 0.507 |
| MDE | 1.3–1.5% → ≤0.25% (about 6×) |
| Labeling turnaround | 2 days → 2 hours (more than 20×) |
| Per-label cost | $0.10 → $0.00002 (down 99.98%) |
The 4B and 8B models perform nearly identically, so 4B wins on cost; both clearly beat a text-only XLM-RoBERTa (QWK +18.2%). The crux is the paired-difference error: single-group query-level error has a slight positive bias, but it all but vanishes in treatment-control paired differences, which is exactly why this is safe for A/B testing. Since launch, relevance-measurement job volume has grown more than 4×. Multilingual markets (French, German, Portuguese) show lower correlation than English (Kendall τ 0.36 to 0.43) but paired differences stay tight around zero.
For anyone running online experiments, the point is not "how accurate is the VLM at relevance" but "what experimental design does cheap labeling unlock." A single VLM judgment is wrong about 17% of the time, so absolute accuracy is not the headline; but A/B testing cares about treatment-control paired differences, the paired design cancels the model bias, and stratification plus larger N drive the MDE down. The model is the means; experiment sensitivity is the payoff. This is also a clean example of LLM/VLM-as-judge working for online evaluation: it is far steadier in a paired setting that is sensitive to differences than in one that is sensitive to absolute scores.
Multilingual relevance is clearly weaker than English, a gap the authors acknowledge but do not diagnose beyond its size. A QWK of 0.507 is only "good," and the small single-group query-level error is real, so the method is not fit to use as an absolute score and depends on the paired design to work at all. The paper is a deployment and engineering study, not a new model architecture (the authors say so themselves), so its transferability hinges on having a comparable volume of human labels to fine-tune on. There is no public dataset to reproduce on.