Qwen3.8-Flash-Next NVFP4 on one RTX 5090: 3 inference engines benchmarked across full 262K context
giveen · reddit · 2026-10-23
A Reddit user benchmarked three inference engines running Qwen3.8-Flash-Next with NVIDIA NVFP4 experts on a single RTX 5090 (32GB): basalt, capy-flash and ninfer-ext, all under identical conditions — full 262,144-token context swept in 16k steps, MTP speculation on, int8 KV cache, zero prefix-cache hits, 256 output tokens per request.
Results:
- basalt wins across the board: 8,490 tok/s prefill and 242 tok/s decode at 16k context, still 8,285/176 tok/s at 255k.
- capy-flash is second (6,921/198 at 16k), ninfer-ext slowest (4,473/92).
- Rankings hold under both greedy decoding (temp 0) and sampled decoding (temp 1.0, topp 0.95, topk 20).
- Full engine configs disclosed: elastic vs fixed int8 KV, MTP spec windows, CPU-side threading, VRAM and host RAM footprints.
More from Infra
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23
- Jensen Huang on NeoCloud financing worries: high demand solves everything else — inductionheads · 2026-10-23