vLLM adds Rubin dual-die localization: pairing MoE weight shards with local compute on both dies
anbarth · x · 2026-11-03
With CUDA 13.4, vLLM can co-locate weights and the compute that reads them: GreenContext(smpartition=...) controls which compute die kernels run on, while LocalizedAllocator controls where their memory is allocated.
Rubin packs two compute dies into one GPU package, so a model can fit but still pay for reads across the NV-HBI bridge. vLLM's optional MoE path pairs weight shards with local compute on BOTH dies — using only one leaves cores idle. Actual vLLM code (localization.py) is linked in the thread.
Related event: vLLM Adapts to Rubin Dual-Die GPUs with Partition-Aware Scheduling(3 posts)→
More from Infra
- Bittensor Subnet Engy Undercuts CoreWeave Inference Pricing by Up to 89% — markjeffrey · 2026-11-03
- Prefrontal open-sources LSA: 72.4x faster attention prefill at 10M tokens, accuracy on par — jcrwhittington · 2026-11-03
- DeepSeek-V4.1-Flash Throughput Benchmarked Across 3 Tools on 4x RTX PRO 6000 — TheZachMueller · 2026-11-03
- My Local AI: free iOS app runs local models on-device with MLX, no account needed — TheOyinbooke · 2026-11-03
- GPU Thermal Throttling Slashes LLM Benchmark Throughput 31% on a GTX 1650 — the-random-guy-2002 · 2026-11-03
- RAM Shortage Cartoon Goes Viral: Greedy Gamers Gulp Supply, Data Centers Left Thirsty — BLUECOW009 · 2026-11-03