Hy AMA reveals Hy4 Flash runs on 96GB VRAM + 128GB RAM, plus QAT quantization details
allegedlyalienated · reddit · 2026-11-21
Practical takeaways from the Hy team AMA on running Hy4 Flash locally:
Hardware: 96GB VRAM + 128GB RAM suffices — keep weights and attention on GPU, offload expert layers and Engram tables to system RAM; Engram params aren't counted in the 12B active, each layer has its own n-gram table.
Quantization: down projections get more bits due to activation outliers; bit-widths chosen via Hessian-based sensitivity; per-tensor allocation preferred over per-expert mixed precision; router trained during full-parameter QAT (no LoRA experiments). As active params approach 3B, reasoning depth, long-context tracking and tool-call reliability degrade first; 2-bit PTQ breaks well before ternary QAT.
Engines: llama.cpp support landing 'in the near future'; close collaboration with vLLM and SGLang.
Related event: Tencent Hunyuan Team Details Hy4 and Flash Models in Reddit AMA(4 posts)→
More from Infra
- AI gateways are becoming the key control layer in production AI stacks: 8 capabilities — goyalshaliniuk · 2026-11-21
- Crowds line up to try local AI demos at NVIDIA RTX Spark launch — JFPuget · 2026-11-21
- Inkling-2-Mini quants are a trap: 17GB weights overflow 48GB VRAM at long context — EmilPi · 2026-11-21
- Cloudflare launches multimodal model clairic-flash, trending on Hugging Face — Cloudflare · 2026-11-21
- Inkling-2-Mini runs on laptops with as little as 9GB RAM, per Unsloth local guide — NielsRogge · 2026-11-21
- Only when infra resources burn out will the application layer see its spring — yangyi · 2026-11-21