Inkling 2 Mini hits ~200 tok/s on a single RTX 4090 with FP8 + NVFP4 quantization

eesahe · reddit · 2026-11-21

A Redditor got Thinking Machines' new Inkling 2 Mini running on a single RTX 4090 at 190-220 tok/s, up from 117 out of the box. Recipe: small vLLM patches (4090 not yet supported for this model), NVFP4 experts (W4A16), FP8 W8A16 attention/head/embeddings, and FP8 KV cache. Fits in 24GB with 35k context; sample-prompt perplexity loss 0.4%. More testing to come.

Original post →

More from Infra

Infra channel →