Deploying 304B Model on Dual DGX Sparks: Extreme Memory Optimization
StartupTim · reddit · 2026-08-08
A developer detailed their experience deploying DeepSeek-V4-Flash-0731 (304B MoE) on 2x NVIDIA DGX Sparks (128GB unified memory each), seeking advice to free up OS RAM.
Current Setup & Bottlenecks
- Hardware & Model: 256GB total unified memory across two nodes connected via ConnectX-7. Native FP8 weights (167GB) with 1M context support.
- Serving Stack: Custom vLLM build with TP=2 and speculative decoding (80% acceptance rate).
- Performance: 82 tok/s decode, 1400 tok/s prefill, passes 146K needle test.
- Memory Allocation: Weights/graphs take 87-89GB, KV cache 11.2GB, leaving only 5-7GB for the OS (dips to 2-3GB under load).
Ruled-Out Solutions
- Lowering GPU memory utilization or max context: Causes vLLM to crash or frees no memory.
- Container caps or enabling Swap: Triggers reclaim loops or system livelocks.
- CPU offload or disk tier: Incompatible with unified memory or FP8 layout.
The Question
The author is looking for actionable ways to reduce vLLM's host-process footprint (API server + workers RSS) and tune NCCL buffers specifically for ConnectX-7 on unified memory systems.
More from Infra
- Does more SMs improve GPU training performance? Stas Bekman explains with numbers — StasBekman · 2026-08-08
- OpenRelay Launches Unified Inference Endpoint: 8 Accelerators, Up to 20% Cheaper — ycombinator · 2026-08-08
- Musk's SpaceX to Build 10GW Nvidia GPU Cluster by 2027, Consuming 30% of Rubin Output — zephyr_z9 · 2026-08-08
- Are Modal and Daytona Pricier Than AWS EC2? Devs Complain About Usability Tax — Pavel_Asparagus · 2026-08-08
- Can a Single RTX 5090 Run MiniMax-H3 Locally? — StartupTim · 2026-08-08
- Deep Dive: Autoscaling Strategies for Peaky LLM Inference Workloads — zainhas · 2026-08-08