Inkling-2-Mini ported to a single RTX 3090: NVFP4 weights, 128k context, 147 tok/s with DSpark
Material_Tone_6855 · reddit · 2026-11-21
A community port of Inkling-2-Mini (NVFP4/MXFP8 checkpoint, no re-quantization) now runs on a single RTX 3090 via a fork of vLLM PR #69028. Key changes: PDL gated to SM90+, no TMA below SM90, Triton fallback for convmix, software-emulated FP8 (E4M3) KV cache verified against torch.float8e4m3fn, and DSpark speculative decoding (7 speculative tokens).
Single-card results: 112–147 tok/s decode with DSpark (91 without), 2,500–2,700 tok/s prefill up to 33k, 131,072-token context, KV pool of 207k tokens across 8 concurrent requests, 17.55 GiB weights with CPU-offloaded embeddings costing only 1% speed. Tool calling works; vision tower disabled and quality benchmarks not yet re-run. VRAM sits at 23.8/24 GB.
More from Infra
- Together AI SVP Simran Arora to headline AI Dev NYC 2026 on AI as its own performance engineer — DeepLearningAI · 2026-11-21
- OpenAI's token efficiency obsession hints launches now wait for serving costs, not research readiness — VraserX · 2026-11-21
- Inside China's SERES super factory: an industrial brain orchestrates 3,000 robots — HaydnBelfield · 2026-11-21
- 36M-param open-source MCP server lets LLMs watch videos locally on a laptop CPU — Safe-Escape197 · 2026-11-21
- GPU rental prices keep rising: payback in 2.3-2.9 years at 75% utilization — BenBajarin · 2026-11-21
- Hyperscaler capex vastly outstrips AI lab revenue, sparking market skepticism debate — BenBajarin · 2026-11-21