capy-flash v0.7 runs a 125B MoE in 64GB RAM, hitting 234 tok/s on two 16GB cards
FeydRowan · reddit · 2026-10-22
Open-source inference engine capy-flash (v0.7.1) targets small consumer setups:
- Qwen3.8-Flash-Next (125B MoE) now runs with 64GB RAM plus one or two 16GB GPUs: routed experts use ISTA-DASLab's 3-bit GSQ-RCO quants (48GiB in RAM, hot experts in VRAM, rest computed on CPU), peaking at 57GB RAM. Speeds: 121 tok/s on one RTX 5070 Ti, 234 tok/s with two cards via tensor parallelism.
- Qwen3.8-27B 3-bit fits a single 16GB card (13.9GB VRAM) with 64k context, vision and MTP — 174 tok/s single, 288 tok/s dual.
- New AVX2 CPU support (Zen 2/3, Intel client), a fix for the Windows/WSL2 page-locked memory limit, and default guards against empty replies.
- Quality: the 27B 3-bit drops MMLU-Pro from 0.78 to 0.72 (NVFP4 still better with two cards); Flash-Next's 3-bit experts match NVFP4.
The author notes single-GPU users may prefer Strata for Flash-Next today; capy-flash shines with two cards thanks to tensor parallelism and a shared VRAM expert cache. One-line install script and quantized weights are on Hugging Face.
More from Infra
- SGLang adds live GPU scaling for MoE deployments without restarts via Elastic EP + NIXL — ying11231 · 2026-10-22
- Modal CEO Erik Bernhardsson on why AI compute shouldn't be sold in 3-year rectangles — Latent Space · 2026-10-22
- US announces $389M investment in advanced chips for fusion, AI data centers — Polymarket · 2026-10-22
- Qdrant Facet API: building accurate search sidebar counts — qdrant_engine · 2026-10-22
- Kangwook Lee details inference setup: W4A16 quant, ~120 tok/s single-stream without drafter — Kangwook_Lee · 2026-10-22
- Neoclouds hit $9B quarterly revenue, up 223%, but can't sell to the $400B enterprise market — DavidLinthicum · 2026-10-22