TensorSharp hits 4.4–5.5x llama.cpp prefill throughput on DeepSeek V4.1 Flash with 5x A40

fuzhongkai · reddit · 2026-10-23

The TensorSharp developer benchmarked his engine against llama.cpp running DeepSeek-V4.1-Flash-1017 (Unsloth GGUF, 300.5GB) on five NVIDIA A40s over PCIe, same files and machine.

Key findings:

With the default YOCO optimization (layers 20–39 process only the last 140 positions, per DeepSeek's DeepInfer approach), TensorSharp reached 4.4–5.5x llama.cpp's prefill throughput on real-text prompts (960.5 vs 174.6 tok/s at 1.8K tokens); with YOCO off it still leads 2.4–3.2x.
Plain decode gains are modest (1.08–1.20x); the DSpark speculative drafter pushes decode to 1.25–1.33x.
Quality trade-off measured on 23,030 WikiText-2 tokens: YOCO raises perplexity 0.67% vs TensorSharp itself (+1.14% vs llama.cpp with YOCO on), concentrated in the first few hundred generated positions.

The writeup honestly flags limitations: perplexity deltas don't map to reasoning/coding ability, and GPU peer access failed on this PCIe machine.

Original post →

TensorSharp hits 4.4–5.5x llama.cpp prefill throughput on DeepSeek V4.1 Flash with 5x A40

More from Infra