Deep Dive: Hybrid Deployment of vLLM and Ray Across Heterogeneous Hardware
jwhh91 · reddit · 2026-07-26
A user discusses deploying large models using vLLM and Ray in a complex heterogeneous setup: two DGX Sparks and an x64 box with an RTX 5090, networked via ConnectX-7.
The goal is to run a 300B (150GB) MoE model keeping TP=2 on the Sparks, while shifting attention heads and hot experts to the 5090's GDDR7. They ask what it would take to augment Ray in vLLM to achieve this compute shifting. The author also shares positive experiences using ConnectX-7 and NVME RAID1 for model weight loading at 20 Gb/s.
More from Infra
- Three chokepoints dominate the global chip supply chain: Taiwan, ASML and Spruce Pine — sahilypatel · 2026-07-26
- Kibble adds offline dataset building and local RAG for any OpenAI-compatible backend — ioncehackedmyschool · 2026-07-26
- A DRY sampler can erase speculative decoding gains on Gemma 4 31B, benchmark finds — Krasnopjorovs · 2026-07-26
- A new report argues storage is becoming a core layer in agentic systems — BenBajarin · 2026-07-26
- LLM config tuning thread weighs KV cache tradeoffs in an 80K-context Laguna setup — kingo86 · 2026-07-26
- SK Hynix says HBM is not the final answer to AI’s memory-wall bottleneck — rwang07 · 2026-07-26