capy-flash v0.16 runs Inkling 2 Mini on two 16GB GPUs at 149 tok/s with 254k context in NVFP4
FeydRowan · reddit · 2026-11-22
Developer Valerio Dolci released capy-flash v0.16, adding Inkling 2 Mini support to his inference engine (alongside Qwen3.8-27B and Qwen3.8-Flash-Next), optimized for tensor parallelism on two 16GB Blackwell cards on Linux; single 5090 and Windows via WSL2/Docker should work but are untested.
Benchmarks (2× RTX 5070 Ti 16GB, from the infographic):
- 149 tok/s chat decode, 194 tok/s code decode, 5,032 tok/s prefill on 4k prompts
- 254k context fully in VRAM — no 1M context because weights, activations and KV cache all stay unquantized in original NVFP4, and the model's global attention makes offloading KV to RAM pointless
- Long-context decay is steep: decode falls to 53 tok/s and prefill to 1,413 tok/s at 250k
The author plans targeted quantization for speed after measuring current performance. Code and weights are open on GitHub and Hugging Face.
More from Infra
- Uncensored 26.4B Inkling 2 Mini runs locally on a 12GB laptop, keeps 98% MMLU — udmrzn · 2026-11-22
- Metal NVFP4/MXFP8 Kernels Cut Inkling 2 Mini Memory from 46.2GiB to 9.9GiB on M3 Max — GabGarrett · 2026-11-22
- Training compute cost per task may be mere cents, argues 1a3orn — if models generalize from small runs — 1a3orn · 2026-11-22
- Miners 'hacked' Chutes for a year with millions in incentives to forge real end-to-end TEE privacy — const_reborn · 2026-11-22
- lxask: a fine-tuned 1.5B model turns plain English into offline Linux commands, self-checked against man pages — Arch-r007 · 2026-11-22
- Second OpenRouter Leak in a Month: MiniMax-Route Prompts May Have Been Read — const_reborn · 2026-11-22