Questions on vLLM MoE Memory Offloading
MelodicRecognition7 · reddit · 2026-07-17
The author attempted to run a roughly 156GB MoE model on a machine with 96GB VRAM + 256GB RAM. Despite trying vllm's --cpu-offload-gb 256 and reducing --max-model-len to 32768, they still encountered torch.OutOfMemoryError: CUDA out of memory.
They pose two questions:
- Is this hardware configuration still insufficient to support this model with a 32k context length?
- Or does the vllm RAM offload feature simply not work properly for this model or MoE architectures in general?
The core issue boils down to: Where exactly are the practical boundaries between CPU offload, VRAM, and context length when running large MoE models locally?
More from Infra
- NeurIPS 2026 workshop will focus on on-device intelligence and local execution — YiMaTweets · 2026-07-21
- How to build a PostgreSQL-backed semantic search pipeline with pgvector and Ollama — KhuyenTran16 · 2026-07-21
- NeurIPS 2026 workshop calls papers on on-device intelligence — YiMaTweets · 2026-07-21
- Milled from Solid Aluminum: AI Rig Multi-GPU Case for Local Compute — dee_hw · 2026-07-21
- FutureCaribbean’s Buildathon offers $50K, H200 compute, and an NYSE pitch — HeyAmit_ · 2026-07-21
- A new series tests which data-science workflows can run on GPUs today — pandeyparul · 2026-07-21