AWS shows how to run prefill/decode disaggregated LLM inference on EKS with multi-node Ray LLM DLC

AWS ML Blog · rss · 2026-10-12

AWS ML Blog walks through deploying a single LLM across two GPU nodes on Amazon EKS using the new multi-node Ray LLM Deep Learning Containers, exposing one OpenAI-compatible endpoint with disaggregated prefill/decode inference.

Key points:

Why PD disaggregation: prefill is compute-bound while decode is memory-bandwidth-bound; sharing one GPU lets long prompts stall ongoing generation. Separating the phases lets each run on tuned hardware and scale independently — well suited to RAG, summarization, and agentic workloads.
What the DLC ships: vLLM as the engine, Ray Serve's HTTP layer, NIXL for cross-node KV-cache transfer, and the EFA stack (libfabric + aws-ofi-nccl), with all components version-validated together to avoid drift.
Architecture: an EKS cluster in private subnets; a CPU system node (m7i.xlarge) runs the Ray head and KubeRay operator, while two g6.8xlarge GPU nodes run the prefill and decode deployments. KV cache flows via NIXL's libfabric backend over EFA, so the GPU node group must stay in one Availability Zone.
Minimal config: the whole serving app is a RayService YAML block using the built-in buildpdopenaiapp helper with prefill/decode configs and NixlConnector settings — no handler classes or custom images.

Complete code is available in the accompanying repository.

Original post →

AWS shows how to run prefill/decode disaggregated LLM inference on EKS with multi-node Ray LLM DLC

More from Infra