TensorSharp Hits 500+ tok/s Prefill for DeepSeek V4.1 Flash on 8×A40 GPUs
fuzhongkai · reddit · 2026-09-13
The open-source TensorSharp inference engine added a dedicated DeepSeek V4.1 Flash path, reaching 533–539 tok/s prefill and 40 tok/s single-stream decode on 8× NVIDIA A40s (Q2K), and 451–492 tok/s prefill with 48.9 tok/s aggregate at concurrency 4 (Q4KM), with 65K context and F16 KV cache. Key findings: keeping 60 GiB of quantized Engram tables GPU-resident boosted prefill from 210 to 535 tok/s; consolidating backends cut decode scheduler splits from 570 to 8; async host warming reduced host MoE offload from 3 layers to 1; and routed-expert TP=8 was slower than layer split (22 vs 32 tok/s) without NVLink, as cross-NUMA communication dominated. Author invites discussion on Engram-style lookup, large-MoE placement, and batching.
Related event: DeepSeek Ships V4.1-Flash, Compressing KV Cache to 890 Bytes per Token(6 posts)→
More from Infra
- AI spending spree deepens free cash flow holes; Microsoft stays positive via $25B bond offering — Kr00ney · 2026-10-31
- Google beams Earth prompt to TPU on orbiting satellite, proving space AI data centers work — ns123abc · 2026-10-31
- Google Cloud spend caps go GA with API, live cost reports and automatic monthly resets — rseroter · 2026-10-31
- Swift 1.5 Flash Next Heretic quants add IQ4_XS and IQ3_S tuned for reverse engineering — jjusko20 · 2026-10-31
- ClickHouse discloses unauthenticated RCE vulnerability CVE-2026-116366, patches available — Hacubu · 2026-10-31
- Count document passes, not tokens: batching questions cuts LLM scoring cost — petrusenko_max · 2026-10-31