Deploying 304B Model on Dual DGX Sparks: Extreme Memory Optimization
StartupTim · reddit · 2026-08-08
A developer detailed their experience deploying DeepSeek-V4-Flash-0731 (304B MoE) on 2x NVIDIA DGX Sparks (128GB unified memory each), seeking advice to free up OS RAM.
Current Setup & Bottlenecks
- Hardware & Model: 256GB total unified memory across two nodes connected via ConnectX-7. Native FP8 weights (167GB) with 1M context support.
- Serving Stack: Custom vLLM build with TP=2 and speculative decoding (80% acceptance rate).
- Performance: 82 tok/s decode, 1400 tok/s prefill, passes 146K needle test.
- Memory Allocation: Weights/graphs take 87-89GB, KV cache 11.2GB, leaving only 5-7GB for the OS (dips to 2-3GB under load).
Ruled-Out Solutions
- Lowering GPU memory utilization or max context: Causes vLLM to crash or frees no memory.
- Container caps or enabling Swap: Triggers reclaim loops or system livelocks.
- CPU offload or disk tier: Incompatible with unified memory or FP8 layout.
The Question
The author is looking for actionable ways to reduce vLLM's host-process footprint (API server + workers RSS) and tune NCCL buffers specifically for ConnectX-7 on unified memory systems.
More from Infra
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16
- DeepSeek V4.1 Flash on 4x DGX Spark: weights alone are 511GB, the 494 tok/s claim doesn't fit — Aroochacha · 2026-10-16
- Open-sourced Intel GPU Vulkan inference upgrades: 2x ingestion, 60%+ faster inference — Fun-Aardvark-1143 · 2026-10-16
- ELSFP pain points: >3dB insertion loss, costly PMF, harsh packaging requirements — jwt0625 · 2026-10-16