Benchmarking 7 Qwen3.8 FN inference engines on a single RTX 5090: Basalt and Infernix lead
jesdga95 · reddit · 2026-10-24
The developer of Basalt benchmarked seven Qwen3.8 FN inference engines (Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp, Basalt) on one RTX 5090 at 400W with 128GB RAM, each converting NVIDIA's NVFP4 weights with its own tool, greedy sampling, int8 KV cache, cold starts. Key findings:
- Decode: Basalt and Infernix lead; Basalt is 10% faster on prose and conversation (354 vs 349 tok/s on struct is a tie). capy-flash is closest of the rest but is built mainly for two-GPU tensor parallelism.
- TTFT/prefill: tied at 25K (3.94s), Basalt pulls away from 50K up—23.79s vs 31.81s at 200K context; prefill reaches 8,666 tok/s at 100K.
- llama.cpp ran without MTP, so its decode (39 tok/s) is reference only.
Method: wikitext prefixes of 25K-200K tokens then a 512-token essay; gaps under 5% treated as ties. Full commit/weight-conversion/drafter configs included.
More from Infra
- Miles v0.1.3 cuts first RL weight update from 99.5s to 3.9s, shares GPUs across train/rollout — hsu_byron · 2026-10-24
- 98%+ Tensor Core utilization shown in LLM training run, drawing admiration — gharik · 2026-10-24
- We spent millions on AI apps with zero revenue — but our old GPUs are now worth more than we paid — stephen280up · 2026-10-24
- Polymarket puts 24% odds on AI data centers launching to space by end of next year — Polymarket · 2026-10-24
- Polymarket jokes wolves are entering Finnish villages after Google cleared forest for AI data centers — Polymarket · 2026-10-24
- The most concerning AI bubble signals: idle GPUs, data center debt, and concentrated revenue — The AI Daily Brief · 2026-10-24