capy-flash runs Qwen3.8-Flash-Next 125B in NVFP4 on two 16GB consumer GPUs
FeydRowan · reddit · 2026-10-16
capy-flash, an evolution of the ninfer-tp2 engine, runs Qwen3.8-Flash-Next 125B-A6B in tensor parallel across two RTX 5070 Ti 16GB cards with experts offloaded to RAM, using NVIDIA's official NVFP4 checkpoint. One flag switches back to Qwen3.8-27B.
- Performance: 100 tok/s generation vs 78 for Strata on the same NVFP4 weights; 3.4k tok/s prefill on a 15k prompt vs 2.7k. Strata's 3-bit IQ3S is still faster at generation but writes 23% more tokens per task
- Quality verified against a datacenter NVFP4 reference via GSM8K, MMLU-Pro, and agentic coding tasks
- Side finding: with tensor parallel, each GPU rounds partial sums to BF16 before the all-reduce, so TP=2 rounds twice — large errors occur 5x more often than on one GPU. Writing partial sums in FP16 (same bytes, 8x finer) restores single-GPU parity at no speed cost
- Limits: Linux, RTX 50 series (sm120) only. Repo and weights are open
More from Infra
- Reflection open-sources 501B MoE Beam; vLLM day-0 support hits 708 tokens/s — Xianbao_QIAN · 2026-10-16
- Vercel Sandbox Drives go GA: attach TBs of data in sub-second, fork instantly — cramforce · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16