Running 285B DeepSeek-V4-Flash-Vision on 12x RTX 3090s at 120+ tok/s
ciprianveg · reddit · 2026-09-09
Developer ciprianveg published a fully reproducible setup for running DeepSeek-V4-Flash-Vision-Exp (285B MoE) on consumer Ampere GPUs:
- FP4 experts + FP8 attention, 157 GB weights, SM86-compatible vLLM build
- 60+ tok/s decode on 10x RTX 3090 (TP2xPP5, 240W cap) with DSpark speculative decoding (k=3); 120+ tok/s on 12 GPUs (TP4xPP3)
- Vision, tool calls and spec decoding all working; 1M context without offload / 4M with RAM offload, 3,500 tok/s long-context prefill
Ships with a pre-built Docker image ghcr.io/ciprianveg/3090-vllm:dsv4-flash-vision-sm86 and a GitHub repo with build guide, start scripts and runtime patches covering DSpark propose-gate (spec+vision fix), scheduler mm×spec row-crossing, grammar-bitmask validation, ViT OOM fix, FlashInfer workspace-lane keying and more.
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- SpaceX locks in $40B GPU financing deal with Apollo, the largest ever committed — TechCrunch AI · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27
- vLLM eyes Rust migration as inference engines race to rewrite the stack — AccBalanced · 2026-10-27