vLLM gains RAM offloading: DeepSeek-V4-Flash-Vision-Exp runs on 4x R9700 locally
sloptimizer · reddit · 2026-09-30
Reddit user sloptimizer reports that thanks to tcclaviger's contribution, vLLM now supports RAM offloading, making frontier models far more accessible on local setups. The author ran the original DeepSeek-V4-Flash-Vision-Exp on four AMD R9700 GPUs.
The post includes a full working podman command with notable settings:
- --tensor-parallel-size 4 with --enable-expert-offload --expert-offload-mem 160 to offload expert weights to RAM
- --kv-cache-dtype fp8, --block-size 256, --max-model-len 256000 for memory-efficient long context
- dspark speculative decoding (3 tokens) for faster inference
- prefix caching, chunked prefill, auto tool choice, and deepseekv4 parsers
More from Infra
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16