Snap distills LLM relevance judges into small models for ad retrieval offline evaluation
_reachsumit · x · 2026-10-14
Snap's arXiv paper proposes an offline user-product relevance metric for evaluating Dynamic Product Advertising (DPA) retrieval systems, addressing sparse/biased engagement signals and costly online A/B tests.
How it works:
- Compresses engagement history into a compact set of intent representatives via Semantic-ID-grouped, event-stratified segmentation
- Uses LLMs as semantic relevance judges to score each recommended product against each intent, aggregating into a user-level relevance score
- An agentic, human-aligned prompting framework generates multi-category relevance guidelines, and teacher judgments are distilled into an efficient student model for scale
The metric correlates strongly with offline and online business metrics, enabling faster, more reliable retrieval iteration before A/B testing.
More from Research
- AIPOCH's Open-Science ships portable .science packages that share full research context, not just outputs — ahuja_priyank · 2026-10-14
- Video-Arena: Game-Based Video Benchmark Shows Native Models Score Just 13.2 F1 — Kuan Zhang · 2026-10-14
- LSCBench: Vibe Coding Agents Write Functional Code, But Only 25% Is Also Secure — Qwen · 2026-10-14
- Omni-LLMs Know More Than They Say: NAVER's OmniCatalyst Adds 7.29% via Self-Generated Text — naver-ai · 2026-10-14
- Repeated-Data Pretraining Study: Smaller Batches Act as Regularizers and Beat More GPUs — Sewon Min · 2026-10-14
- NAVER's WATCH: Cross-Video Human-Video Pretraining Lifts Robot Success Rate to 77.7% — naver-ai · 2026-10-14