Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context
TumbleweedNew6515 · reddit · 2026-11-03
A detailed writeup of running DeepSeek-V4.1-Flash locally on four Tesla V100 SXM2 32GB cards (2017-era) with 256GB DDR4, using the 1Cat-vLLM fork for Volta support, a 232GB IQ2XXS GGUF, a 110GB Engram table in RAM, and 1/3 of expert weights offloaded to pinned host memory.
Benchmark results:
- Decode: 36-38 tok/s from 8K to 75K context, nearly unchanged with thinking on/off
- Prefill: 1,000-1,200 tok/s up to 128K depth
- 524,288-token context window, 5-token speculative decoding (DSpark drafter, FP8)
- Decode drops to 26 tok/s before the Engram file is pre-read into page cache
Hardware details: full NVLink mesh, SM clocks locked at 1522MHz, 150-175W per card, 22 min model load from SATA. Next test: 8x V100 setup.
More from Infra
- A single B300 now matches the world's fastest 1997 supercomputer in FP64 FLOPs — scaling01 · 2026-11-03
- Anthropic IPO: investors push for $1.5T-$1.6T valuation, below ~$2T discussed — rohanpaul_ai · 2026-11-03
- Cerebras engineer explains wafer-scale cooling: water flows straight into the wafer — bookwormengr · 2026-11-03
- Goertzel counters Altman's idle-RAM take with decentralized compute networks — bengoertzel · 2026-11-03
- Satire: companies borrow a 'quadrillion' to build 420 data centers in 6 months — asymco · 2026-11-03
- SF billboards slam cloud AI for stealing data; blogger predicts decade of on-device iPhone assistants — brandon_xyzw · 2026-11-03