TensorSharp ships Day-1 MiniMax-M3.1-Flash support: 9.8× faster 64K prefill than llama.cpp on 8× A40
fuzhongkai · reddit · 2026-10-25
The TensorSharp developer added local inference support for MiniMax-M3.1-Flash on day one of its open-weight release and benchmarked it against Unsloth's llama.cpp minimax-m31 branch on 8× A40 GPUs (same UD-Q4KXL GGUF, 232.3 GiB total). Prefill throughput hit 1,699/2,449/2,567 tok/s at 8K/32K/64K — 1.3×/3.6×/9.8× faster — cutting 64K time-to-first-token from 293.4s to 40.0s. Decode gains are smaller (1.49× at 64K; short-chat nearly identical). The gap stems from MiniMax Sparse Attention: TensorSharp runs block-sparse CUDA kernels while the llama.cpp branch uses dense masks. It also supports GGUF, vision input via an OpenAI-compatible API, JSON/schema output, tool calling, multi-turn KV-cache reuse, tensor parallelism (41 tok/s short-chat decode, unsupported in the llama.cpp branch), DSpark speculative decoding with confidence gating (32.3 vs 25.8 tok/s), and MoE expert CPU offload (79.7 GiB in host memory, 17.9 vs 13.8 tok/s).
More from Infra
- Jensen Huang unveils Vera Rubin DSX POD at GTC Berlin, pitches Europe's AI economy — nvidia · 2026-10-25
- ANEMLL Forge 0.2.1 runs Qwen3.8-27B at up to 192K context on M6 Mac Mini's Neural Engine — Competitive-Bake4602 · 2026-10-25
- Compute contracts vs reality: Nscale has only 5% of contracted GPUs live, TeraWulf 12% — AccBalanced · 2026-10-25
- Live MCP rewrite from Node to Rust cuts memory usage by roughly 8x — draginol · 2026-10-25
- 305M-Parameter Sparse MoE Runs Computer-Use Agent Fully Offline on a MacBook Air — lmoroney · 2026-10-25
- SGLang reruns DeepInfer on matching hardware, matches or beats its prefill performance — hsu_byron · 2026-10-25