Inkling 2 Mini quantized to 13.9GB runs at ~60 t/s decode on a single RTX 4090
indrasmirror · reddit · 2026-11-21
A developer got Thinking Machines' open-weights Inkling 2 Mini — a 26B looped MoE (3B active, 12 core layers run 3 passes per token) — running on one RTX 4090 after abliteration and quantization to a 13.9GB GGUF.
- Built a custom engine, Xolotl, porting the graph from Pedro Cuenca's llama.cpp PR and fixing a loop bug (FP4 scales missing on passes 2 and 3)
- Benchmarks: 59.7 t/s decode, 1,162 t/s prefill at 8K; held-out KL vs. parent just 0.013 with 96.3% top-1 agreement
- Refusal rates: 4/100 on harmful prompts, 1/100 on harmless ones
- MTP drafter and vision projector released as optional files; kernel-fusion work underway targeting 100+ t/s decode
Weights on Hugging Face, engine on GitHub.
More from Infra
- Optical waveguide crossing lets light pass through while keeping circuits electrically isolated — jwt0625 · 2026-11-21
- Sentdex: RTX PRO 6K stock returning and prices falling as loopholes close — good for consumers — Sentdex · 2026-11-21
- Son approached Wall Street to fund a $1T SK Hynix buyout, per Bloomberg scoop — tengyanAI · 2026-11-21
- RTX Spark desktops swap 200G ConnectX-7 for 10GbE, killing two-box clusters for large MoEs — AIFlow_ML · 2026-11-21
- Early Inkling 2 Mini LoRA tests: same results with under a third of the energy — ziqiao_ma · 2026-11-21
- AMD AI DevDay Paris on Dec 12: vLLM Maintainer to Talk Decode Optimization, Free ROCm Cert — AccBalanced · 2026-11-21