Edge inference with llm-d: prefill-decode disaggregation on Qualcomm Cloud AI 100 Ultra

HowDevelop · x · 2026-11-16

A LearnKube Day lightning talk on edge AI inference with llm-d explains why prefill (one big parallel compute pass, compute-hungry) and decode (a token-by-token loop, memory-bandwidth-bound and latency-sensitive) should run in separate pods scaled independently, so long prefills don't stall decode on shared instances. It includes a documented NeuReality deployment: Meta Llama 3.1 70B served with vLLM in an 8-prefill + 1-decode disaggregated pattern, with Qualcomm Cloud AI 100 Ultra cards joined as worker nodes via Qualcomm's Kubernetes device plugin.

Original post →

More from Infra

Infra channel →