Edge inference with llm-d: prefill-decode disaggregation on Qualcomm Cloud AI 100 Ultra
HowDevelop · x · 2026-11-16
A LearnKube Day lightning talk on edge AI inference with llm-d explains why prefill (one big parallel compute pass, compute-hungry) and decode (a token-by-token loop, memory-bandwidth-bound and latency-sensitive) should run in separate pods scaled independently, so long prefills don't stall decode on shared instances. It includes a documented NeuReality deployment: Meta Llama 3.1 70B served with vLLM in an 8-prefill + 1-decode disaggregated pattern, with Qualcomm Cloud AI 100 Ultra cards joined as worker nodes via Qualcomm's Kubernetes device plugin.
More from Infra
- Peter Diamandis: The Country Building Gigawatts Fastest Will Set the Price of Intelligence — PeterDiamandis · 2026-11-17
- Gemini Managed Agents Adds Self-Hosted Sandboxes: Google Runs the Loop, You Run the Tools — _philschmid · 2026-11-17
- RWKV-8 RPU preview hits 12,500+ tokens/s real-time inference — kastnerkyle · 2026-11-17
- Your AI budget is too generous and too restrictive at once: routing saves, flat caps kill rewrites — ruslansv · 2026-11-16
- AI SRE that closes the deployment loop: OTel analysis, Argo Rollouts canaries and AI remediation — HowDevelop · 2026-11-16
- Bindu Reddy: China lacks Blackwells, US lacks power — an AI slowdown looks inevitable — bindureddy · 2026-11-16