When LLM-Inferred User Context Adds Value in Production Streaming Recommendation
Milad Sabouri, Neeraj Sharma, Sardar Hamidian, Shaghayegh Agah
cs.IR
2026-09-30
On Comcast streaming, LLM-generated profiles lose 30.7% Recall@10 to centroid profiles overall, win only for the 18% exploratory users, and cut catalog coverage by about 80%.
Context in recommender systems has mostly meant a short list of predefined variables: time of day, device, location. The ceiling is obvious; most factors that shape what someone watches can't be enumerated in advance or observed directly. LLMs offer an alternative: feed the interaction history plus item text to a model, get a natural-language summary of the user's preferences, encode it, and use it in place of the conventional profile that averages item embeddings into a centroid.
The prior evidence comes from public e-commerce benchmarks, where an attention-fused LLM profile beat aggregate profiles on a movie/TV catalog but lost on games. Those catalogs ran from a few thousand to about 14,000 items, evaluation measured accuracy alone, and the conditionality sat between datasets. This paper moves the question onto a production streaming platform: roughly 240,000 items ranked in full, 10,000 real users, beyond-accuracy metrics, and for the first time the conditionality is examined at the level of individual users.
A 2×2 design crosses representation type (aggregate vs. LLM-generated) with contextual scope (full history vs. a short/long split fused by attention):
Everything downstream is held fixed: a frozen SBERT encodes item titles and descriptions, a two-layer MLP scores the concatenation of profile and item vectors, and training uses binary cross-entropy with five uniform negatives per positive. The profile vector is the only experimental variable.
Users are split post-hoc into regimes: an Explorer has no test-period item anywhere in the training history; everyone else is a Non-Explorer. The sample holds 1,761 Explorers, about 18%.
At PCT = 25%, all numbers are percentage change against Centroid within the same segment and cutoff; marks significance at α = 0.05:
| Strategy | Recall@10 All | Recall@10 Explorer | Coverage@10 All | Novelty@10 All |
| Narrative | -30.7% | +18.7% | -79.8% | -17.8% |
| TemporalNarrative | -21.6% | +14.3% | -79.2% | -19.9% |
| TemporalCentroid | +0.5% | -7.4% | -5.8% | +1.9% |
On the full population both LLM profiles lose across the board; Narrative is down 26.9% on NDCG@10 and every comparison is significant. Habitual users are about 82% of the sample, so the aggregate numbers largely are their numbers. On Explorers the ordering flips: Narrative is up 31.5% on NDCG@10 (significant) and 18.7% on Recall@10, though HitRate misses significance at both cutoffs, which the authors attribute to the smaller subsample.
The beyond-accuracy effects are the most consistent in the paper. LLM profiles lift within-list diversity by about 4%, cut catalog coverage by roughly 80%, and lower novelty by 18–22%, pushing recommendations toward popular items. The authors call this a popularity-attractor effect, and every per-user diversity and novelty comparison is significant across both cutoffs and all three segments. The short-term window matters too: on Explorers, TemporalNarrative drops 9.5% at PCT = 15% and 19% at PCT = 30%, with 20–25% the stable band.
The regime is partially predictable from training-period features: history size correlates with the Explorer label at r = -0.30 and within-history content diversity at r = +0.11 (both p < 0.001). Users with shorter, more varied histories explore more.
This is a rare controlled answer to whether LLM-generated profiles earn their keep in production, and the answer is: not for most users. When the regime is unknown, defaulting to the aggregate profile is the conservative choice; capturing the Explorer gain requires solving a separate prediction problem before serving. There is also a trap easy to miss: selecting profiles by regime doesn't remove the popularity bias, it concentrates that bias on exactly the exploratory users a system would otherwise rely on to broaden exposure.
For practitioners the boundary is more useful than the method: a centroid sits close to already-consumed items and suits habitual viewers, while a thematic summary reaches beyond the history and suits users whose next choices depart from it.
The authors list: one platform, one content domain, one generation-and-encoding pipeline; the finding is itself conditional and may condition differently elsewhere; whether the earlier domain-level differences reduce to regime composition is untested; and the evaluation is offline while the selection mechanism would run online.
Points that deserve a further discount: the Explorer definition folds cold starts and genuine taste changes into one label; r = +0.11 is far from a usable predictor; the popularity-attractor explanation does not separate generation from encoding and is labeled a hypothesis; and HitRate on Explorers is not significant, so the conclusion on the critical 18% of users is not fully stable.