NInfer-all adds Inkling 2 Mini, nearly 2x llama.cpp decode speed on RTX 5090

WaveCut · reddit · 2026-11-21

The author added Thinking Machines' Inkling 2 Mini (26B MoE, 3.4B active) to NInfer-all, a multi-arch inference fork that runs Unsloth's UD-IQ4XS GGUF without requantization, with vision and the DSpark drafter working.

Benchmarks vs llama.cpp on the same GGUF (Unsloth build, since mainline can't load it yet):

On cards with 16GB or less, experts live in system RAM with the GPU caching hot ones; even 8GB works thanks to MoE, and experts can stream from disk (9 tok/s on the 5060). Prebuilt binaries for Windows/Linux (CUDA 12/13) and Docker are available; the repo also supports Qwen3.6/3.8, Gemma 4, DeepSeek-V4.1-Flash and more, with measurements across 25 cards.

Original post →

More from Infra

Infra channel →