Inkling-2-Mini ported to a single RTX 3090: NVFP4 weights, 128k context, 147 tok/s with DSpark

Material_Tone_6855 · reddit · 2026-11-21

A community port of Inkling-2-Mini (NVFP4/MXFP8 checkpoint, no re-quantization) now runs on a single RTX 3090 via a fork of vLLM PR #69028. Key changes: PDL gated to SM90+, no TMA below SM90, Triton fallback for convmix, software-emulated FP8 (E4M3) KV cache verified against torch.float8e4m3fn, and DSpark speculative decoding (7 speculative tokens).

Single-card results: 112–147 tok/s decode with DSpark (91 without), 2,500–2,700 tok/s prefill up to 33k, 131,072-token context, KV pool of 207k tokens across 8 concurrent requests, 17.55 GiB weights with CPU-offloaded embeddings costing only 1% speed. Tool calling works; vision tower disabled and quality benchmarks not yet re-run. VRAM sits at 23.8/24 GB.

Original post →

More from Infra

Infra channel →