TensorSharp ships day-1 MiniMax-M3.1-Flash support with 9.8x faster 64K prefill vs llama.cpp
fuzhongkai · reddit · 2026-10-25
The developer of TensorSharp, a native .NET LLM inference engine and agent runtime, added MiniMax-M3.1-Flash support on day 1 of its open-weight release and published benchmarks on an 8×A40 host using Unsloth's UD-Q4KXL GGUF (232.3 GiB), against Unsloth's llama.cpp minimax-m31 branch.
Key numbers: 64K executor prefill hit 2,567 tok/s vs 262 tok/s (9.8x); server time-to-first-token on a 64K prompt dropped from 293.4s to 40.0s; 32K prefill 2,449 vs 681 tok/s; 64K decode 22.9 vs 15.4 tok/s; short-chat decode roughly tied. An agentic tool loop ran 18.1s vs 20.1s, dropping to 11.7s with tensor parallelism (41 tok/s short-chat decode); both engines reused 99% of the prompt on follow-up turns.
The gain comes from MiniMax Sparse Attention (16×128-token blocks per KV group beyond position 2,048): TensorSharp runs block-sparse CUDA attention directly, while the tested llama.cpp branch uses a dense mask with dense flash attention, plus multi-GPU prompt-processing pipelines. Supports OpenAI-compatible serving, structured output, parallel tool calls, conversation-cache reuse, multi-GPU layer splitting/tensor parallelism, DSpark speculative decoding and CPU expert offload.
More from Infra
- CrowdGPT: a crowd-trained, datacenterless 1B open-source LLM, 6.9B tokens done — Vxtzq1 · 2026-10-25
- Apples-to-apples rerun: DeepInfer tops DeepSeek's published throughput by ~9% on 4x B300 — charles_irl · 2026-10-25
- Apollo's viral "circular AI spending" charts debunked: Ramp card data skews to seed startups — 2C_ornot2C · 2026-10-25
- Open source and local AI as a counterbalance is dawning on people — mjdramstead · 2026-10-25
- OpenClaw creator burned $10 million on AI tokens in a single month — Scobleizer · 2026-10-25
- rakyll ships aic, a Docker-like container tool built for AI developers — rakyll · 2026-10-25