Multimedia Asset Personalization via Multimodal Embeddings at Netflix
Emma Yanyang Kong, Aditya Deshpande, Bowei Yan, Asad Abbasi, Santiago Castro, Avneesh Saluja, David Fagnan, Ashish Rastogi
cs.IR
2026-08-19
CLIP lets Netflix collapse five artwork models; short-panel IPS rises 5.7%. Tri-modal MediaFM then lifts preview streaming +0.193% versus ID-only.
Netflix picks artwork tiles and autoplaying preview clips per member. Older models treated each asset as an ID. They could not see who is in the frame, what the palette is, or what the tone is. New titles and new assets have almost no clicks, so the system falls back to popularity. Five canvases (billboard, vertical box, horizontal panel, short panel, landscape) each had their own model, so signal on high-impression surfaces never reached the others.
This paper is not a new algorithm. Concatenating a frozen embedding into a two-tower item tower is standard. The contribution is the production path: a shared Embedding Store, a fully precomputed request path, and a cheap proxy that screens embeddings before an A/B. The authors also write down the failure modes.
Artwork: CLIP's 768-d image vector is concatenated with the learned ID embedding inside the item tower and projected back by an MLP. The user tower and retrieval stack stay put. Five per-canvas models collapse into one. Canvases differ in impression volume and in what counts as a positive, so pooling raw logs would let the busiest canvas dominate. Each example is weighted by the long-term reward of its interaction type. Low-volume canvases then contribute by how much the action is worth, not how often it fires.
CLIP's joint space also gives search for free. The homepage score is mixed with cosine between the query text vector and the artwork image vector. The mixing weight α is tuned online. No new model, no new embedding.
Previews: an earlier content-aware baseline, SeqCLIP, encodes frames with a CLIP-like model and averages them. No audio, no dialogue, no order. MediaFM is an in-house tri-modal model pretrained on catalog shots. Each shot concatenates three frozen encoders: visual SeqCLIP, audio wav2vec 2.0, timed text from text-embedding-3-large, into 2304-d, then a BERT-style Transformer trained with masked shot modeling. A preview vector is the mean of its shots taken in the full-title context, which beats encoding the clip alone.
Three precompute layers keep latency flat. Embeddings are written at asset ingest. Item-tower outputs are cached per asset. Daily materialization covers each profile's top 500 titles; the rest fall back to unpersonalized popularity. That head is most of what members see and cuts precompute cost by up to 50%. Offline metrics use IPS on explore traffic with propensities logged exactly, not estimated.
Before any new embedding is plumbed through, a linear probe on content vectors alone tries to pick each title's popularity winner under uniform serving. Clear losers are dropped before IPS and A/B spend.
Artwork ablations, offline IPS versus the old per-canvas models:
| Canvas | CLIP only (V1) | Unify only (V2) | Unify+CLIP (V3) |
| Short panel | +1.065% | +2.243% | +5.691% |
| Landscape | +0.421% | +1.654% | +1.667% |
| Horizontal | −0.027% | +0.245% | +1.087% |
| Billboard / vertical box | flat to slightly down | flat to slightly down | roughly flat |
A/B across devices for at least four weeks: V1 −0.0265% on the discovery metric (n.s.), V2 +0.0236% (n.s.), only V3 +0.127% (p<0.005). During the Eclipse TV UI launch, which shifted the dominant canvas from vertical box to short panel, a holdback showed +0.233% discovery and +0.184% streaming hours. Extrapolated against 191 billion hours in 2025, that is about 350 million extra hours per year. Query-aware artwork on Search added +0.36% playthrough (p<0.05).
Previews versus ID-only: SeqCLIP +0.332% offline IPS and +0.187% online streaming; MediaFM +0.380% / +0.193%. On the proxy, MediaFM beats pick-at-random by 25.90 points, SeqCLIP by 18.75, matching the online order.
What travels is three engineering choices, not a new fusion block. Keep embeddings on a platform so a new foundation model is a registration, not a rewrite. Run nothing on the request path, so latency SLAs do not move for multimodal features. Probe candidates before spending member traffic. For a team that already has two-tower retrieval, the cheapest change is a frozen vector in the item tower, plus honest canvas mixing.
The hard lesson is complementarity. CLIP alone and unification alone do not move the online metric. Together they do. High-volume canvases teach the shared item tower how those image dimensions map to taste. Cold-start canvases and brand-new assets can use that mapping on day one.
Online lifts sit in tenths of a percent; they look large only at Netflix scale. V1 and V2 being non-significant means "add content features" can be called a failure when the model lacks the data to use them. MediaFM's online edge over SeqCLIP is 0.006 points. On previews where the picture already tells the story, audio and timed text are redundant; the authors say the gain concentrates on tone-ambiguous titles, especially TV autoplay. The proxy predicts an unpersonalized winner and is not a personalization metric. It is a rejector. The appendix notes that MediaFM inherits whatever frozen encoders drop, and fusion is not learned jointly. Only 500 head titles are materialized daily; the long tail still gets popularity.