Flyweight: open-source C++/CUDA engine runs VRAM-busting MoE models on one GPU plus system RAM
Main-Wolverine-1042 · reddit · 2026-09-18
Flyweight (Apache-2.0) is a native C++/CUDA GGUF inference runtime that runs MoE models bigger than your VRAM on a single consumer NVIDIA card plus system RAM, auto-planning expert offload at startup. On a 5070 Ti 12GB laptop it hits 35 tok/s decode on Qwen3.8-Flash-Next IQ1S, 40 tok/s on a 27B IQ2XXS, 6-7 tok/s on DeepSeek-V4-Flash, and runs Z-Image-Turbo beside the chat model. Kernels compile at runtime via NVRTC (no CUDA toolkit in wheel), KV cache supports f16/q80/4-bit TurboQuant, tool calls are grammar-enforced, and it ships OpenAI/Anthropic-compatible APIs plus a chat UI. Author openly used Claude Code for much of the coding, with parity-tested kernels.
More from Infra
- Google, Nvidia and Anthropic want to unlock 100 GW for AI — power may be the real bottleneck — TansuYegen · 2026-09-18
- Huawei Moves Ascend 960DT AI Chip Launch Up to Early 2027 — emmanuelvivier · 2026-09-18
- Subsidised coding subscriptions will shrink; policy-based token routing is coming — craigbalding · 2026-09-18
- Tencent's video model demo now runs locally: 7.5GB encoder cut and an MLX branch — gaganghotra_ · 2026-09-18
- AI ported a distro in 20 minutes — is NVIDIA sawing off its own CUDA moat? — BringTea_666 · 2026-09-18
- India Lands $12 Billion in Semiconductor Investment Pledges Within Months — pstAsiatech · 2026-09-18