Hy AMA reveals Hy4 Flash runs on 96GB VRAM + 128GB RAM, plus QAT quantization details

allegedlyalienated · reddit · 2026-11-21

Practical takeaways from the Hy team AMA on running Hy4 Flash locally:

Hardware: 96GB VRAM + 128GB RAM suffices — keep weights and attention on GPU, offload expert layers and Engram tables to system RAM; Engram params aren't counted in the 12B active, each layer has its own n-gram table.

Quantization: down projections get more bits due to activation outliers; bit-widths chosen via Hessian-based sensitivity; per-tensor allocation preferred over per-expert mixed precision; router trained during full-parameter QAT (no LoRA experiments). As active params approach 3B, reasoning depth, long-context tracking and tool-call reliability degrade first; 2-bit PTQ breaks well before ternary QAT.

Engines: llama.cpp support landing 'in the near future'; close collaboration with vLLM and SGLang.

Related event: Tencent Hunyuan Team Details Hy4 and Flash Models in Reddit AMA(4 posts)→

Original post →

More from Infra

Infra channel →