Inkling 2 Mini hits 613 tok/s decode and 1M context on one RTX 5090 via Basalt
jesdga95 · reddit · 2026-11-21
Thinking Machines' small MoE model Inkling 2 Mini, which outperforms Qwen 3.8 27B on some tasks, was benchmarked by a Reddit user running it on a single RTX 5090 with Basalt 0.5.5.
- Speed: 287 tok/s prose, 613 tok/s structured, 338 tok/s chat; 14.1K tok/s prefill at 2K context — versus 95 decode/3.1K prefill for llama.cpp IQ4NL GGUF on the same card
- Quality preserved: NVFP4 experts kept as released, KLD 0.003, 97.8% top-1 match
- 1M context on one 5090 with cold experts spilling to RAM; needle tests pass at 131K and 400K; 60 tok/s on a 16GB 5060 Ti
- 3.4B active params, competitive on agentic coding; supports DSpark drafting, image input, tool calls
- Note: garbage output with NVFP4 GGUF was a llama.cpp loop slot bug; Unsloth's fix PR is merged
More from Infra
- SpaceX lenders resist $40B of AI chips in space: collateral would be unrecoverable — basedjensen · 2026-11-21
- 801-30B-A3B tech report: decomposed pretraining self-improvement loops on 11.8T tokens — jiqizhixin · 2026-11-21
- NVIDIA paper: frequent context compaction cuts LLM serving cost up to 7x vs long context — khademinori · 2026-11-21
- Analyst speculates OpenAI's problems stem from GPU shortage, no SpaceX access — mark_k · 2026-11-21
- Researcher builds local GPU server to run 24/7 AI agent system after Mac mini runs out of memory — nodosenlared · 2026-11-21
- Custom engine runs Inkling 2 Mini at 129.5 tok/s with 344K context on a single RTX 3090, 2.3x faster than llama.cpp — teortaxesTex · 2026-11-21