Running a 295B Model on 16GB VRAM
antirez · x · 2026-07-10
A developer demonstrated how to run Tencent's Hy3, a 295B parameter MoE model, on a single RTX 4060 Ti 16GB GPU. Using NeutronStar (based on the author's CUDA branch ds4), attention layers and shared experts are kept in VRAM, while routed experts are streamed from the SSD on a per-token basis. This enables interactive chat with a massive model on a roughly $400 GPU.
The implementation adds a GQA attention path and utilizes a native 2-bit GGUF format for ds4, achieving speeds of about 1.8 tok/s. The author also provided a link to the model weights.
More from Infra
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27