Flyweight: open-source C++/CUDA engine runs VRAM-busting MoE models on one GPU plus system RAM
Main-Wolverine-1042 · reddit · 2026-09-18
Flyweight (Apache-2.0) is a native C++/CUDA GGUF inference runtime that runs MoE models bigger than your VRAM on a single consumer NVIDIA card plus system RAM, auto-planning expert offload at startup. On a 5070 Ti 12GB laptop it hits 35 tok/s decode on Qwen3.8-Flash-Next IQ1S, 40 tok/s on a 27B IQ2XXS, 6-7 tok/s on DeepSeek-V4-Flash, and runs Z-Image-Turbo beside the chat model. Kernels compile at runtime via NVRTC (no CUDA toolkit in wheel), KV cache supports f16/q80/4-bit TurboQuant, tool calls are grammar-enforced, and it ships OpenAI/Anthropic-compatible APIs plus a chat UI. Author openly used Claude Code for much of the coding, with parity-tested kernels.
More from Infra
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Gigabyte's AI TOP motherboard packs 2TB RAM and 9 GPUs for home AI labs — vincent_koc · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- imec neural probe integrates photonics directly into 130nm CMOS backend — jwt0625 · 2026-11-04