Inkling 2 Mini on 2× RTX 3090: 557K Context + Vision at ~100 tok/s via NInfer-all
Sisuuu · reddit · 2026-11-22
A Reddit user details running Thinking Machines' Inkling 2 Mini (26.4B looped MoE, 3.4B active) locally on 2× RTX 3090 without NVLink/P2P. vLLM fails (checkpoints need sm90+, no Ampere expert kernels), but NInfer-all with Unsloth UD-IQ4XS + DSpark drafter hits 557K context at 100 tok/s on two cards (174K at 127 tok/s on one). KV cache is BF16-only (36 KiB/token) and pinned to card 1; --tensor-split 1,127 maximizes context. The real win is RAM prefix caching: a 63K doc prefills once (90s), then a new chat over it starts in 0.85s TTFT. Needle-in-haystack passes at 5K/63K/162K. Full launch command and tuning notes included.
More from Infra
- Nemotron 3.6 Ac Academia Support Lands in llama.cpp: Looped Hybrid with Gated Delta Product — maternion · 2026-11-22
- Sparse MoEs make SSD the last bottleneck: expert streaming breaks the VRAM wall — rickasaurus · 2026-11-22
- OpenAI engineer confirms GPT-Live cache affinity bug, fix rolling out Monday — pbbakkum · 2026-11-22
- Lambda runs Nvidia CPO switches on 10,000+ GPU GB300 NVL72 cluster, cutting latency up to ~10% — Beth_Kindig · 2026-11-22
- Dragos planted fake AWS keys in 2M AI prompts — the keys got used — serendip-ml · 2026-11-22
- Running Inkling 2 Mini locally on Mac: 20GB model, ~16 tok/s on M4 Pro — jacob_posel · 2026-11-22