ICL-Bandit: Using Bandit Selection of Prompt Examples to Improve LLM Ad Relevance Labeling
burkov · x · 2026-11-07
Judging whether an ad answers a user's query normally requires costly, slow expert labeling, while LLMs often miss the specialized context needed. ICL-Bandit treats example selection for the prompt as a repeated selection problem: it tries different demonstrations, checks whether the model's label matches a known expert label, and uses that feedback to improve future selections.
- Evaluation used a demonstration pool of 1.58M expert-labeled examples
- Time-ordered train/test splits of 10,000 and 2,000 query–ad pairs
- The approach generalizes to other in-context-learning labeling tasks needing curated demos
More from Research
- MCPHallu: 14 frontier MCP agents average 0.642, only 23% admit infeasible goals — dair_ai · 2026-11-07
- Nature Cancer's DeepFAN: AI nodule reading aids judged by how they change radiologists — jiqizhixin · 2026-11-07
- Once AI can simulate a human cell, aging becomes a software problem — rand_longevity · 2026-11-07
- Is generative drift a fundamental flaw in AI? Reddit debates agent reliability — Peacefulhuman1009 · 2026-11-07
- ICLR authors: rebut nitpicking reviewers by quoting their exact claims — brunofmr · 2026-11-07
- CIKM 2026 tutorial on Agentic Search covers RL optimization, deep research, and evaluation — mrdrozdov · 2026-11-07