Inkling 2 Mini on 2× RTX 3090: 557K context + vision at ~100 tok/s with NInfer-all
Sisuuu · reddit · 2026-11-22
A detailed hands-on report of running Thinking Machines' Inkling 2 Mini (26.4B looped MoE, 3.4B active) locally on 2× RTX 3090 without NVLink/P2P:
- vLLM fails: official NVFP4/MXFP8 checkpoints need sm90+; after patching load gates, no working Ampere kernel exists for the expert GEMMs.
- NInfer-all works: with Unsloth UD-IQ4XS quants + DSpark drafter + vision, 557K context at 98-104 tok/s on two cards, or 174K at 127 tok/s on one.
- Key insight: KV cache is BF16-only (36 KiB/token) and lives entirely on the first --devices card; only routed experts split across cards. --tensor-split 1,127 plus --vision-residency overlay and --prefill-chunk 1024 maximizes context.
- Prefix caching is the real win: a 63K document prefilled once (90s TTFT) gives 0.85s TTFT on a new chat over the same doc, with 62,973/62,996 tokens served from a 32 GiB host cache.
- Benchmarks: 4K prefill at 2,371 tok/s; needle-in-haystack correct at 5K/63K/162K; defaults proved faster than every attempted tweak. Two-card decode is slower since every MoE block crosses PCIe via host RAM.
More from Infra
- John Langford weighs in on FBT note: fixed-point convergence and quantized caching — JohnCLangford · 2026-11-22
- Fewer than 10 companies can safely move tenants across multi-tenant GPU clusters, argues compute-market skeptic — AccBalanced · 2026-11-22
- Tens of thousands of compute nodes power this multi-agent swarm architecture — hwchase17 · 2026-11-22
- Teknium shares Hermes telemetry: 92.4% of tokens are cache reads — Teknium · 2026-11-22
- Musk demos real-time AI video generation on half of one AI4 TRIP accelerator — jamesdouma · 2026-11-22
- Sentdex asks for ~150 free H100s right after Martin Shkreli posted wanting to buy them — Sentdex · 2026-11-22