Lawyer runs DeepSeek-V4.1-Flash on 8x 2018 V100s: 215 tok/s on code, 524K context fully in VRAM
TumbleweedNew6515 · reddit · 2026-11-06
A lawyer on Reddit (part 4 of a series) got DeepSeek-V4.1-Flash (2-bit GGUF) running on eight 2018 Tesla V100-SXM2-32GB cards, and the results are genuinely usable:
Numbers
- Code: 215 tok/s at 1K prompt, still 183 at 128K and 160 at 256K; legal writing 110-118 tok/s; structured JSON 180-207
- Prefill 2,200 tok/s at 128K; first token at 32K in 17s, 256K in 2 min
- 524K context with all 248.8 GB of weights resident in VRAM — no expert paging from system RAM during decode
How
- 1Cat-vLLM's DeepSeek-V4.1 Volta port, reworked to pipeline parallel 2 × tensor parallel 4 across two NVLink boards talking over PCIe
- Model Runner V2, full CUDA graphs, DSpark drafter guessing 5 tokens per round
- Three days of per-switch tuning (150W power cap, locked clocks), each change gated on bit-for-bit identical A/B output; 30 failed ideas documented in the repo
He estimates 50-60% of existing LLM workflows can run on this setup at speeds matching Claude Code in a chat window, with RAG, tool calls, and a Discord-style SSH interface all wired in. Repo is open-sourced.
More from Infra
- Kimi K3.1 Pro gets Day-0 TPU support: serving a 2.8T-param MoE with llm-d on TPUs and GPUs — vllm_project · 2026-11-06
- Level1Techs: CUDA porting lags while AMD's R9700 turns out surprisingly fast — Thrumpwart · 2026-11-06
- Satirical pitch: cool data centers with fluorinert and market them as 'water-free' — typedfemale · 2026-11-06
- US stocks lost $3.8T in a day; chipmakers took $1.15T of the hit — CharlieMuirhead · 2026-11-06
- 'This is why we need at least a datacenter/home': the case for local compute — tech__unicorn · 2026-11-06
- Reddit hopes for cheap RAM by Christmas as memory prices stay in focus — keepthepace · 2026-11-06