NVIDIA Shows How to Serve 753B-Parameter GLM-5.3 Across Two DGX Stations with vLLM
NVIDIA Developer · youtube · 2026-11-14
NVIDIA published a tutorial on locally serving GLM-5.3, a 753-billion-parameter MoE model whose NVFP4 checkpoint (464 GB) exceeds a single GB300's 252 GB memory.
Method and verification:
- Connect two DGX Stations via ConnectX-8 and split the model with tensor parallelism, keeping all weights in GPU memory with no CPU offload
- RDMA setup: collect network values, download the pinned checkpoint, copy it to the second station, run the same vLLM script on both
- Verification: API health check, RDMA check (GPUs communicating over the network, not TCP fallback), nvidia-smi showing one vLLM worker per GPU
- Benchmarked per-user throughput and latency with NVIDIA AIPerf and SPEED-Bench Coding prompts at concurrency 1/8/16/32
Use cases: sensitive data, local agent prototyping, and per-token-cost-free prompt iteration. The playbook and tech blog include every command.
More from Infra
- MLX over Vulkan hits 124 tok/s on RTX 5060 Ti, beating llama.cpp CUDA by 25% — SumitGup · 2026-11-22
- Intruder swapped Prime Intellect's hosted DeepSeek for an abliterated copy for 53 hours — mawcopolow · 2026-11-22
- Token economics model projects $1.14T global API spend by 2030, but only 1% of users pay beyond coding — AccBalanced · 2026-11-22
- Zero-degradation Strata tuning: fixing layer_split lifts decode ~10-15% and halves CPU expert misses — PathfinderTactician · 2026-11-22
- llama.cpp Vulkan SCALE fix PR #32735 delivers 2.3-3.2x speedup on Radeon R9700 — pmttyji · 2026-11-22
- $1400 decade-old V100 32GB runs Qwen 27B: Q4_K_M +48% gen speed, Q8_0 +16% prefill — PuzzleheadedArm3732 · 2026-11-22