Basalt 0.5.0 runs Tencent Hy4 Flash locally at 172 tok/s on a 5090 with 1M context

jesdga95 · reddit · 2026-11-20

Basalt 0.5.0 adds support for Tencent's newly released Hy4 Flash: 1M context, dynamic vision input, and an NVFP4 single-file pack needing 95–100GB of memory. On a 5090 + 5060 Ti rig it hits 84 tok/s on prose, 172 tok/s on structured output, and 3,450 tok/s prefill.

Early impressions: throughput is lower than Flash Next because Hy4 Flash has nearly twice the weights per token, with more costly cache misses — maybe 10–20% performance still to unlock. On real tasks it's roughly a tie with Flash Next (slightly better at math), but Flash Next is much faster on the same hardware. Quantization hurts it: official Q4KM scored just under 87% top-1 match vs BF16, and the NVFP4 build 86%, versus Flash Next NVFP4 above 91%.

Original post →

More from Infra

Infra channel →