capy-flash runs Qwen3.8-27B at 379 tok/s on two RTX 5070 Ti cards
FeydRowan · reddit · 2026-10-24
Developer releases capy-flash v0.7.4, a self-built inference engine purpose-made for tensor parallelism across two consumer RTX 50 cards over PCIe, running Qwen3.8-Flash-Next and Qwen3.8-27B, with a cross-card VRAM expert cache (misses computed on CPU) tuned for NVIDIA NVFP4 quantization.
Benchmarks (2× RTX 5070 Ti 16GB, Ryzen 9 9900X, 96GB DDR5):
- Flash-Next NVFP4: 178 tok/s decode, 4,828 tok/s 64k prefill; 3-bit experts hit 234 tok/s decode with GSM8K matching NVFP4 and MMLU-Pro dropping from 0.78 to 0.72
- 27B dual-card NVFP4+DFlash2: verified 379 tok/s decode, beating the fastest other LocalMaxxing run (SGLang fork on 2× 5090 at 315); up to 927 tok/s in concurrency tests
- vs Strata: 1.76× faster on average with NVFP4 on two cards (53-turn coding session 177s vs 268s); with same 3-bit experts Strata is faster overall, but mostly due to its route tail skip — disabled, Strata decodes 10% slower on one card
Also supports single-card start (5060 Ti and up), mixed card pairs, and Windows via WSL2/Docker, letting users add a second GPU later without changing engine or model files.
More from Infra
- Chart: Hyperscalers' combined free cash flow projected to collapse near zero by 2026 — SumitGup · 2026-10-24
- Unlocking hidden RTX 4070 Ti SUPER compute: FP16 matmul gets 1.9x faster — lxfater · 2026-10-24
- a16z charts: US electrical equipment capacity down 9% since 2019 as prices jump 69% — a16z · 2026-10-24
- Flux.3 Klein generates 768x1024 images in 15s on an M5 iPad Pro with a 4-step Distill LoRA — JosephJacks_ · 2026-10-24
- Open-source framework streams any MoE model on any device: 35B at 25 tok/s on a 16GB Mac — dai_app · 2026-10-24
- Over a third of AI spend by top 1% of firms already goes to compute — nathanbenaich · 2026-10-24