TensorSharp hits 5.5× llama.cpp prefill throughput on DeepSeek V4.1 Flash across five A40s
fuzhongkai · reddit · 2026-10-24
The developer of TensorSharp, a native .NET LLM inference engine, details optimizations running DeepSeek V4.1 Flash 1017 on five NVIDIA A40s: server prefill throughput reaches 960.5 vs 174.6 tokens/sec for llama.cpp (5.5×) on 1.8K-token prompts, or 563.9 (3.23×) with the approximate YOCO optimization disabled. Decode is 35.5 tok/s (1.20×), 55.0 with Unsloth's Q80 drafter (1.33×). Key changes: desynchronized expert parallelism using CUDA events and shared-expert overlap; attention kernel reusing key tiles across heads plus local cache mirrors to cut cross-GPU transfers; and YOCO bounded replay running layers 20–39 over only the final 140 positions (toggleable via TSDSV41YOCOKEEP=0), with cache-rewind bookkeeping to re-prefill skipped positions. Both engines used identical Unsloth UD-IQ2XXS GGUF files over PCIe.
Related event: TensorSharp hits up to 5.5x llama.cpp prefill throughput on DeepSeek(2 posts)→
More from Infra
- NVIDIA ships BioNeMo Inference Runtime v0.2 with RFD4-Proteina support — AllThingsApx · 2026-10-24
- ISTA-DASLab ships GSQ-RCO quantized DeepSeek-V4.1-Flash GGUFs, llama.cpp support still pending — Gold_Coconut9777 · 2026-10-24
- Sauers_ Mocks OpenAI Over Unstudied Multi-Agent Hardware — Sauers_ · 2026-10-24
- Local quant benchmark: new releases trail months-old DeepSeek/Qwen models — EmilPi · 2026-10-24
- OpenAI red team finds unauthenticated RCE in widely deployed Brotli compression library — gdb · 2026-10-24
- Meta's Muse Spark Open hits ~150M tokens/s per MW on NVIDIA GB300 in early benchmarks — ArtificialAnlys · 2026-10-24