Spotify details LLM podcast recommendations: SGLang multi-item scoring cuts per-candidate request fan-out
ying11231 · x · 2026-10-24
Spotify Engineering explains how it moved LLM-powered podcast recommendations from proof of concept to production:
- Its generative retrieval architecture, based on Semantic IDs and the PLUM line of work, uses beam search to explore multiple token-sequence paths for candidate generation in real time.
- For candidate ranking, naive fan-out (one request per candidate, then extracting logprobs) gets expensive fast. Spotify instead uses SGLang's Score API with multi-item scoring and candidate masking: the shared prefix (system prompt + user profile/history) is packed once with KV caching, each candidate attends only to itself and the prefix, and label log-probs are read at item boundaries to score and rank all candidates in a single forward pass.
- The feature was built with contributions from LinkedIn's engineering team; Spotify is also collaborating with NVIDIA on TensorRT-LLM for inference.
A practical blueprint for anyone serving LLM-based ranking or recommendation at scale.
More from Infra
- Used NVIDIA DGX Spark reselling at $6,200, well above its ~$4,000 retail price — AIFlow_ML · 2026-10-25
- SGLang Matches DeepInfer on Prefill, Adds 7.4% Decode Throughput on 64×GB300 — BanghuaZ · 2026-10-25
- Poldi: A Fully Offline Local Voice AI Assistant That Runs on 8GB VRAM — rhinodevil · 2026-10-25
- kalomaze says he no longer has meaningful GPU access, back to toy baselines — kalomaze · 2026-10-25
- Is the explosion of LLM inference engines healthy competition or fragmentation? — Anstellos · 2026-10-25
- Jev ecosystem explodes: 5 open-source apps from a 7.1s flight-search agent to SQL-in-English — udmrzn · 2026-10-25