An Empirical Study on Zero-Data Bootstrapping for Conversational Recommender Systems
Rohan Surana, Junda Wu, Zhouhang Xie, Yu Xia, Nathan Kallus, Julian McAuley
cs.IR
2025-04-22
UCSD and Netflix bootstrap CRS from reviews with no in-domain dialogues: GPT-4o synthesizes chats, and Qwen2.5-1.5B Full-SFT raises Recall@1 207.8% over zero-shot on INSPIRED.
Conversational recommender systems stall on data. A new domain usually has reviews, item metadata, and click logs, and almost never has recommendation dialogues. Labeling chats is expensive, privacy rules bite, and knowledge graphs are not a given. Large models can talk zero-shot, but cost, privacy, and interpretability still push many deployments toward smaller in-house models. Those models need fine-tuning, and fine-tuning needs dialogues.
UCSD, Netflix, and Cornell tighten this into zero-data CRS: no in-domain conversational corpus, no external knowledge graph, only reviews, metadata, and collaborative-filtering signals. The question is whether those non-conversational signals can be turned into usable conversational supervision.
The pipeline has three stages: pick seeds, synthesize dialogues with a teacher, fine-tune the target.
Each seed (review text, optional metadata, optional collaborative features) is embedded from the target model's last layer. Generation has not happened yet, so there are no labels and uncertainty sampling does not apply. Two label-free rules compete. Jensen-Shannon (JS) diversity clusters the embedding space and picks high-entropy items that sit far from the selected set, chasing coverage. Fisher information does greedy D-optimal design under a last-layer linearization, chasing the direction that most shrinks the parameter confidence ellipsoid. Both iterate to budget B.
For every chosen item the teacher (GPT-4o) draws 5 Reddit-style movie-recommendation templates and 3 reviews, writes a user query, then writes 20 pseudo target recommendations. B items times K queries give BK training pairs at 2BK offline teacher calls. Selection itself is one embedding pass.
Targets are Llama3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen3-4B, under both LoRA and Full-SFT. Seeds come from Amazon Reviews '23 Movies & TV. Evaluation is ReDial (10,006 dialogues, 2,001 test) and the smaller INSPIRED (1,001 / 200). Baselines: popularity, zero-shot prompting, naive GPT synthesis without domain seeds, and the neighborhood method NBCRS.
Domain-grounded synthetic data beats zero-shot and naive GPT synthesis. Recall is in percent.
| Model | Setting | ReDial R@1 / R@5 | INSPIRED R@1 / R@5 |
| Qwen2.5-1.5B | zero-shot | 2.39 / 6.96 | 0.64 / 1.43 |
| Qwen2.5-1.5B | naive GPT | 2.56 / 7.63 | 0.76 / 1.89 |
| Qwen2.5-1.5B | Synth-LoRA | 3.39 / 9.27 | 1.17 / 3.21 |
| Qwen2.5-1.5B | Synth-SFT | 3.36 / 9.38 | 1.97 / 4.58 |
| Qwen3-4B | zero-shot | 7.09 / 19.58 | 5.42 / 10.20 |
| Qwen3-4B | Synth-LoRA | 8.39 / 23.72 | 5.82 / 10.48 |
| Qwen3-4B | Synth-SFT | 7.73 / 22.81 | 6.01 / 11.35 |
| Llama3-3B | zero-shot | 1.57 / 5.56 | 2.04 / 5.00 |
| Llama3-3B | Synth-SFT | 1.91 / 6.54 | 2.61 / 5.75 |
| NBCRS | raw seed to synth dialogue | R@5 0.39 to 11.52 | not listed |
On INSPIRED, Qwen2.5-1.5B Full-SFT raises Recall@1 by 207.8% over zero-shot; naive GPT only gets +18.8%. On ReDial the same model gains +41.8% (LoRA) and +40.6% (SFT) at Recall@1; the larger Qwen3-4B gains +18.3% and +9.0%. Smaller models absorb synthetic data harder. Llama3-3B LoRA on ReDial lands at 1.53 Recall@1, slightly below the 1.57 zero-shot number; Full-SFT is what actually moves it.
Two annotators scored 100 synthetic query-list pairs near 4 out of 5 on naturalness, coherence, relevance, and diversity. Catalog hit rate is 92.3% for naive GPT and 76.5% for the grounded corpus; the higher-overlap set is the worse teacher. Matching popular titles is not the same as useful supervision.
Active selection beats random and review-count popularity at a fixed teacher budget. At full budget on Qwen2.5-1.5B, ReDial NDCG@5 is 5.82 (random), 7.40 (JS), 7.47 (Fisher). Fisher leans long-tail: at budget 1k, tail items are 61.9% of the pick, Spearman ρ with review count is -0.172. On ReDial it lifts head / tail Recall@5 from 9.44 / 3.70 (random) to 12.10 / 4.44. Metadata and collaborative signals each add structure to selection, more so at larger budgets.
When real chats are scarce, synthetic data can beat them. INSPIRED has about 1k dialogues; synthetic-only training beats the original INSPIRED train set, and mixing the two rises further. ReDial already has broad coverage, the original train set stays competitive, and mixing synthetic in dips slightly.
A new-domain CRS usually ships with reviews and interaction logs, not dialogues. This paper lays out a reproducible cold-start path: pick seeds with information-theoretic rules, have a teacher rewrite non-conversational signals into chats, then fine-tune a model you can host. Selection happens before generation, so what you save is GPT-4o calls.
The paper does not invent a new active-learning criterion. The contribution is running JS versus Fisher, metadata versus CF, and LoRA versus Full-SFT inside one strict zero-data setting. Domains that already hold large dialogue logs should not dump synthetic data in by default.
Evaluation is movies only. Music or e-commerce would need new seeds, templates, and benchmarks; cross-domain transfer is untested. Synthetic chats can hallucinate and carry teacher style. Relevance is the weakest human score, because the teacher sometimes stuffs weakly related titles. Prompts were not optimized. On mid-resource ReDial, mixing synthetic with real data slightly hurts, and the paper gives no filter. Llama3-3B LoRA on ReDial does not beat zero-shot, so synthetic supervision is not a free win for parameter-efficient fine-tuning.