Basalt 0.5.0 runs Tencent Hy4 Flash locally at 172 tok/s on a 5090 with 1M context
jesdga95 · reddit · 2026-11-20
Basalt 0.5.0 adds support for Tencent's newly released Hy4 Flash: 1M context, dynamic vision input, and an NVFP4 single-file pack needing 95–100GB of memory. On a 5090 + 5060 Ti rig it hits 84 tok/s on prose, 172 tok/s on structured output, and 3,450 tok/s prefill.
Early impressions: throughput is lower than Flash Next because Hy4 Flash has nearly twice the weights per token, with more costly cache misses — maybe 10–20% performance still to unlock. On real tasks it's roughly a tie with Flash Next (slightly better at math), but Flash Next is much faster on the same hardware. Quantization hurts it: official Q4KM scored just under 87% top-1 match vs BF16, and the NVFP4 build 86%, versus Flash Next NVFP4 above 91%.
More from Infra
- LexiEngine: Rust inference engine for AMD runs 27B at full 262k context 47% faster than llama.cpp — W61k3r · 2026-11-20
- Anthropic would need $602B annual revenue—17x growth—to match tech peers' returns — GaryMarcus · 2026-11-20
- Cost chart: Claude Fable 5.2 burns $54 per task, OpenAI a fraction — haider1 · 2026-11-20
- Redditor builds 1.75TB local AI archive to hedge against cloud restrictions — nfored · 2026-11-20
- Anthropic reportedly migrated to AMD GPUs in 3 days, vs 6-12 months previously — AccBalanced · 2026-11-20
- Quantum computing buzz at SC: author ran an X gate on a real QPU, 3-5 years to commercial use — StasBekman · 2026-11-20