Inkling 2 Mini on 2× RTX 3090: 557K Context + Vision at ~100 tok/s via NInfer-all

Sisuuu · reddit · 2026-11-22

A Reddit user details running Thinking Machines' Inkling 2 Mini (26.4B looped MoE, 3.4B active) locally on 2× RTX 3090 without NVLink/P2P. vLLM fails (checkpoints need sm90+, no Ampere expert kernels), but NInfer-all with Unsloth UD-IQ4XS + DSpark drafter hits 557K context at 100 tok/s on two cards (174K at 127 tok/s on one). KV cache is BF16-only (36 KiB/token) and pinned to card 1; --tensor-split 1,127 maximizes context. The real win is RAM prefix caching: a 63K doc prefills once (90s), then a new chat over it starts in 0.85s TTFT. Needle-in-haystack passes at 5K/63K/162K. Full launch command and tuning notes included.

Original post →

More from Infra

Infra channel →