Huawei Cuts Generative Recommendation Inference Latency up to 25x vs vLLM Without Custom Kernels
_reachsumit · x · 2026-11-20
A Huawei paper on beam search for generative recommendation inference shows pipeline-level optimization beats kernel work.
- GR inference differs from LLM serving: long inputs, wide beams, few fixed output tokens. vLLM treats beams as independent batched sequences, causing redundant KV cache reads over the shared prefix, plus per-step engine invocations and PagedAttention block-management overhead.
- Fixes: Tree Attention concatenating beams along the sequence dimension, pre-generated partial attention masks, static KV cache allocation, loop-unrolled generation dispatch, TorchAir graph compilation, operator fusion, vocabulary-constrained decoding.
- Results: up to 25x end-to-end latency reduction vs vLLM on Ascend NPUs, matching a SOTA custom-kernel solution — optimize the whole pipeline before writing custom kernels.
More from Infra
- Anthropic Gets Claude Prefill Running on AMD MI355X in Three Days — AccBalanced · 2026-11-20
- Customer cuts GitHub Actions spend 90%: 20-min job runs in 7 min on Celesto — aniketmaurya · 2026-11-20
- Anthropic's $2T valuation questioned: $104B net loss on $33B revenue in 9 months — deliprao · 2026-11-20
- New video essay traces a prompt's journey through the physical world of AI — soumitrashukla9 · 2026-11-20
- Who is Harvey Jones, the 33-year Nvidia director who shut the door on Huang's return — APPSO · 2026-11-20
- Vercel auto-upgrades 2FA 'sudo mode' sessions to device-bound cookies (DBSC) — cramforce · 2026-11-20