Inkling 2 Mini hits ~200 tok/s on a single RTX 4090 with FP8 + NVFP4 quantization
eesahe · reddit · 2026-11-21
A Redditor got Thinking Machines' new Inkling 2 Mini running on a single RTX 4090 at 190-220 tok/s, up from 117 out of the box. Recipe: small vLLM patches (4090 not yet supported for this model), NVFP4 experts (W4A16), FP8 W8A16 attention/head/embeddings, and FP8 KV cache. Fits in 24GB with 35k context; sample-prompt perplexity loss 0.4%. More testing to come.
More from Infra
- How do you run big batch GPU jobs — thousands of files, frames, or prompts? — mrtonymat · 2026-11-21
- ComfyUI model loading stuck at 400 MB/s even after upgrading to NVMe — C-h_A_o-S · 2026-11-21
- Inkling 2 Mini runs at 125 tok/s on a MacBook, claims best-in-class at its size — simonguozirui · 2026-11-21
- Same RTX 5090, 4x throughput: Surogate hits 2,765 tok/s vs llama.cpp's 683 with 8 users — AIFlow_ML · 2026-11-21
- Compound's AI slowdown scenario: NVDA down ~45% if revenue growth halves and multiples revert — tengyanAI · 2026-11-21
- DatologyAI open-sources Zephon, a data loader with elastic determinism for LLM training — josh_wills · 2026-11-21