Lawyer runs DeepSeek-V4.1-Flash on 2x DGX Spark + 3090s: 2,618 tok/s prefill at 256K, 95.9% KLD accuracy, all data open
TumbleweedNew6515 · reddit · 2026-10-31
Nick Mermiges, the lawyer who previously ran local models on a V100 server, is back: DeepSeek-V4.1-Flash-1017 at 2.75 bpw EXL3 on two DGX Sparks with vLLM tensor parallelism, plus an EPYC 3090 box serving Engram tables over RDMA and running the DSpark speculative drafter on NVLinked 3090s.
- Freed memory goes to a 1.93M-token KV cache plus an NVMe prefix cache: a 150K prompt returns first token in 2.58 s after restart (vs 58.6 s cold)
- Prefill is the standout: 2,827 tok/s at 4K, 2,618 tok/s at 256K — beating MiaAI (2,065–2,155) and bertholomus (2,439–2,568) builds; structured JSON output hits 322 tok/s at 4 streams, code 192 tok/s vs MiaAI's 166
- Weak spots: 66 tok/s single-stream prose (MiaAI: 80.7), 62 tok/s decode after 128K prompts (bertholomus: 99), and a cold 93K prompt blocks short requests for 34 s
- The drafter loves code/JSON (4–5 tokens accepted per round) but manages only 2 on prose
- Accuracy: 95.9% top-token agreement on sudoingX's KLD panel for real tool-call turns (KLD 0.018), versus bertholomus's published 79.4%
Repo, launch scripts, bench tool and raw JSON for every cell are public. Everything was driven through Claude Code, which also ran the benchmarks. Upcoming: full 1M-context test, BFCL tool calling, and head-to-head quant comparisons.
More from Infra
- Community asks: where is the Qwen 4 35B MoE as the Qwen 4 lineup nears — solyarisoftware · 2026-10-31
- Musk predicts orbital compute will outpace Earth's by end of 2028 as SpaceX targets 1M tons to orbit — XFreeze · 2026-10-31
- Koboldcpp v1.124 "aesthetic edition" released for local LLM inference — Fcking_Chuck · 2026-10-31
- CXMT validates G6 DRAM engineering samples in Q3, mass production expected within a year — zephyr_z9 · 2026-10-31
- AI scarcity moved to atoms: memory, GPU and wafer margins hit 58%-80% — sujingshen · 2026-10-31
- Anthropic's S-1 risk factors: compute tops the list at 13.4%, edging out regulation — ayushtweetshere · 2026-10-31