SGLang adds live GPU scaling for MoE deployments without restarts via Elastic EP + NIXL
ying11231 · x · 2026-10-22
LMSYS details Elastic Expert Parallelism Scaling in SGLang with NIXL: MoE deployments can add GPUs in fine-grained increments without restarts or full replicas. A 20-GPU deployment grows to 24 by adding four ranks while existing ranks keep serving. Covers expert-parallel topology expansion, CUDA graph recapture, benchmarks, practical limits, and roadmap (EPLB, DP-attention shards, scale-down). Built with NVIDIA and Mooncake teams.
More from Infra
- Deutsche Bank Deploys Farsight On-Prem as Financial AI Question Shifts to Where Models Run — huggingface · 2026-10-22
- Gradio distills FLUX.3 Klein's 9B prompt upsampler into 0.8B and 2B models that run on CPU — Gradio · 2026-10-22
- Securing coding agents with Docker Sandboxes and edge LLM inference at ATO — HowDevelop · 2026-10-22
- Revolut's massive outage blamed on GCP, but ex-sales contact says it rejected multi-zone redundancy — thedealdirector · 2026-10-22
- 0.9B Model Transcribes a 4-Person Meeting on a MacBook With 0.08% Speaker Confusion — Binary_orchid · 2026-10-22
- Qdrant 1.20 ships BM25 hybrid search without sparse vectors — qdrant_engine · 2026-10-22