TensorSharp hits 41 tok/s decoding DeepSeek V4.1 Flash on 8× A40
fuzhongkai · reddit · 2026-09-12
The developer of TensorSharp, an open-source LLM inference engine, shares optimizations for running DeepSeek V4.1 Flash on 8× NVIDIA A40 with layer-split execution:
| Metric | Optimized result |
|---|---|
| Q2K single-request decode | 41.1 tok/s |
| Q2K prefill (GPU-resident Engram test) | 532–541 tok/s |
| Q2K cold model load (MooseFS) | 144–155 s (2.5× faster) |
Key changes:
- Less synchronization: wrapping CUDA and custom operators into one backend per GPU cut decode graph partitions from 565–577 to 8, lifting decode from 35.6 to 41.1 tok/s (15%).
- GPU-resident Engram: quantized lookup tables stay on GPUs, avoiding CPU row lookups and eliminating a 130-second host-table warmup.
- Sequential loading beats more threads: giving each reader a contiguous file range restored effective readahead on network storage, the main reason for the 2.5× load speedup.
The author notes these are project benchmarks, not a claim of superiority over llama.cpp, since no compatible same-weight reference was available, and welcomes independent runs.
More from Infra
- What actually runs AI models at the edge in 2026: Mac mini, DGX Spark, iPhone 17 Pro — MaziyarPanahi · 2026-09-12
- Running Qwen3.8 Flash Next on dual RTX 3090: full llama.cpp config shared for tuning — ChopSticksPlease · 2026-09-12
- UAE redesigns 5GW AI campus with bunkers and air defenses after Iranian strikes on Gulf cloud facilities — mark_k · 2026-09-12
- DeepSeek V4.1-Flash Runs 502GB Model on a Single RTX 5090 at 5-21 tok/s — AccBalanced · 2026-09-12
- Running 100-200 agents daily: disk space is now the bottleneck, not compute — vincent_koc · 2026-09-12
- Orca releases uncensored MLX weights for DeepSeek V4.1 Flash, cutting refusals by 87-96% — AccBalanced · 2026-09-12