Running a 295B Model on 16GB VRAM
antirez · x · 2026-07-10
A developer demonstrated how to run Tencent's Hy3, a 295B parameter MoE model, on a single RTX 4060 Ti 16GB GPU. Using NeutronStar (based on the author's CUDA branch ds4), attention layers and shared experts are kept in VRAM, while routed experts are streamed from the SSD on a per-token basis. This enables interactive chat with a massive model on a roughly $400 GPU.
The implementation adds a GQA attention path and utilizes a native 2-bit GGUF format for ds4, achieving speeds of about 1.8 tok/s. The author also provided a link to the model weights.
More from Infra
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26
- Ubicloud publishes "Fixing Tool-Call Accuracy in vLLM" — petrusenko_max · 2026-10-26
- Meta approves nonvoting Class C shares as 2026 capex guidance hits $135B–$145B — thione · 2026-10-26
- Unsloth Ships MiniMax-M3.1-Flash Quants: 2-bit Retains 91.5% Accuracy, Runs in 150GB RAM — danielhanchen · 2026-10-26