Running a 295B Model on 16GB VRAM
antirez · x · 2026-07-10
A developer demonstrated how to run Tencent's Hy3, a 295B parameter MoE model, on a single RTX 4060 Ti 16GB GPU. Using NeutronStar (based on the author's CUDA branch ds4), attention layers and shared experts are kept in VRAM, while routed experts are streamed from the SSD on a per-token basis. This enables interactive chat with a massive model on a roughly $400 GPU.
The implementation adds a GQA attention path and utilizes a native 2-bit GGUF format for ds4, achieving speeds of about 1.8 tok/s. The author also provided a link to the model weights.
More from Infra
- NVIDIA publishes Vera CPU architecture details before AMD’s AI event — ryanshrout · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- HilbertRaum open-sources a fully local AI chat and document analysis app for private use — Vladowski · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22