Basalt engine runs Inkling-2-Mini at 613 tps on one RTX 5090, 3x faster than llama.cpp
udmrzn · x · 2026-11-23
The developer behind the Basalt inference engine keeps pushing local LLM speeds:
- Inkling-2-Mini (Thinking Machines' new 26.4B looped-transformer MoE): up to 613 tps structured output, 338 tps chat, 287 tps prose on an RTX 5090; even an RTX 5060 Ti 16GB hits 60 tps. The NVFP4 model pack is just 19.7GB (Basalt 0.5.5).
- Same-card comparison: llama.cpp (IQ4NL GGUF) decodes 95 tps vs Basalt's 287 tps prose — roughly 3x.
- Prompt processing: 14,100 tps at 2K context, 8,900 tps at 128K.
- A day earlier, Basalt 0.5.0 ran Tencent's Hy4-Flash (224B total / 12B active) at up to 172 tps using an RTX 5090 + 5060 Ti with system RAM overflow, via a 159GB NVFP4 pack.
- Basalt supports native vision, DSpark and more.
More from Infra
- French dev laments missed chance to build green data center ecosystem on low-carbon grid — Dimillian · 2026-11-23
- Fchollet: deployment friction may be the best thing for AI infra, stretching the buildout timeline — npinto · 2026-11-23
- Ex-Goldman CEO Blankfein: AI Is Now Disrupting the Earnings-Stock Price Relationship — firstadopter · 2026-11-23
- VoiceShelf ships on Android: offline Kokoro TTS hits 45x realtime via custom NPU runtime — Simple-Lecture2932 · 2026-11-23
- Databricks exec: inference providers routinely botch ZDR commitments — pwendell · 2026-11-23
- Cerebras CEO: architecture choices sidestep HBM, CoWoS and TSMC 3nm bottlenecks — rohanpaul_ai · 2026-11-23