Basalt adds NVFP4: Qwen3.8 Flash-Next on a single 5090 with 8,582 tok/s prefill
jesdga95 · reddit · 2026-10-18
Basalt, an inference engine for Qwen3.8 Flash-Next on Blackwell, landed its top two requested features: native NVFP4 and faster prefill.
NVFP4: new .basalt packs keep stock NVIDIA experts and store dense layers as Q6K. On a 5090 (400W) + 128GB DDR5:
- struct 304 tok/s, prose 183 tok/s; with a peer 5060 Ti: struct 479 / prose 239
- Quality vs Q80 (100 chunks): PPL 3.308 (Q80: 3.286), KLD 0.068, 91.5% same top-1
Prefill: 8,582 tok/s on a 64K prompt (7.6s to first token). Versus Infernix on identical NVFP4 weights and hardware: up to 1.3x decode and 1.6x prefill.
A batch of stability/perf fixes also landed; Unsloth Q4 + Q8 (near-lossless) tuning is next. Code and weights are on GitHub and Hugging Face.
More from Infra
- Modeling tokens-to-value: solving agent harness configs via a single utility function — JoshPurtell · 2026-10-18
- Open-source Intro to Semiconductor Manufacturing book follows a wafer from sand to chip in 25 chapters — CatAstro_Piyush · 2026-10-18
- When every ablation you run costs $1M — gordic_aleksa · 2026-10-18
- DeepSeek-V4.1-Flash on 2x DGX Spark: TensorFold decodes 92 tok/s code, 2.1-2.3x faster than vLLM kit — shayanx45 · 2026-10-18
- MiniCPM-V 4.7 hits HF trending: 35B-A3B MoE runs on phones in ~3GB — lmoroney · 2026-10-18
- AI race won't be won by code but by concrete: power plant permits take 7 years — r0ck3t23 · 2026-10-18