NInfer-all adds Inkling 2 Mini, nearly 2x llama.cpp decode speed on RTX 5090
WaveCut · reddit · 2026-11-21
The author added Thinking Machines' Inkling 2 Mini (26B MoE, 3.4B active) to NInfer-all, a multi-arch inference fork that runs Unsloth's UD-IQ4XS GGUF without requantization, with vision and the DSpark drafter working.
Benchmarks vs llama.cpp on the same GGUF (Unsloth build, since mainline can't load it yet):
- RTX 5090: prefill 6,664 vs 2,318 tok/s, decode 182 vs 97
- RTX 4090: prefill 4,537 vs 2,307, decode 140 vs 75
- RTX 3090: prefill 2,046 vs 1,236, decode 112 vs 60
- RTX 5060 8GB: prefill 1,074 vs 293, decode 33 vs 15
On cards with 16GB or less, experts live in system RAM with the GPU caching hot ones; even 8GB works thanks to MoE, and experts can stream from disk (9 tok/s on the 5060). Prebuilt binaries for Windows/Linux (CUDA 12/13) and Docker are available; the repo also supports Qwen3.6/3.8, Gemma 4, DeepSeek-V4.1-Flash and more, with measurements across 25 cards.
More from Infra
- 26 LLM routers caught injecting malicious tool calls, draining a client's $500k wallet — 0xkarasy · 2026-11-21
- OpenRouter accused of sending users' BYOK private keys to China despite earlier denials — 0xkarasy · 2026-11-21
- OVH's Olala Mini 7A1B: 360 tok/s decode on one RTX 5090, 96K context, v2.0 to exceed 100B — JiliJeanlouis · 2026-11-21
- If you bet the century on AI, you need the financial and physical supply chain to back it — we don't — ZeroStateReflex · 2026-11-21
- Obscura: a 70MB Rust headless browser with zero Chromium that drops into Puppeteer and Playwright — thisguyknowsai · 2026-11-21
- Airbnb reportedly runs 80,000 daily chats on Alibaba's open-weight Qwen as Chinese models undercut US APIs up to 30x — mawcopolow · 2026-11-21