Deco decouples 'thinking' from 'spelling' in generative recommenders for 3.2-8.6x throughput
_reachsumit · x · 2026-10-26
A new arXiv paper introduces Deco (decoding from concepts), tackling the serving cost of semantic-ID beam search in generative recommenders. Standard autoregressive interfaces pay for SID length K twice: history of T items enters the backbone as TK code tokens and each beam-search step runs through the backbone, costing O(BK) decode tokens per user. Deco's backbone condenses history once into a compact next-item concept; a light KV-cached decoder runs the entire beam search. On seven e-commerce, movie and music datasets it matches or beats full-backbone accuracy while delivering 3.2-8.6x throughput under a 50ms p95 latency budget.
More from Infra
- Compute rental rates rise across the curve as AI buildout funding tightens, Meta 10-Q shows — AccBalanced · 2026-10-26
- Open-source recipe: abliterated DeepSeek-V4.1-Flash at ~200 tok/s on 4x RTX PRO 6000 — burny_tech · 2026-10-26
- Micron opens second advanced packaging plant in Houli, a concrete HBM step — tengyanAI · 2026-10-26
- What do you do with the spare 8 GPUs in a 64-card NVL72 setup? Keep them hot as spares — _xjdr · 2026-10-26
- Jalapeño chip is just a systolic array with HBM: why custom inference silicon is easier than you think — ai · 2026-10-26
- Torch.compile Flag Cut 16 Regions to 2, Delivering 4.53x Speedup on B200 — cfregly · 2026-10-26