Ninfer-ext: from-scratch C++/CUDA engine hits 2,142 tok/s decode on Qwen 35B nvfp4 with one RTX 5090
giveen · reddit · 2026-10-14
ninfer-ext is an extended fork of NInfer — a from-scratch C++/CUDA inference engine running one model at a time on a single RTX 5090 (32GB) with no Python in the inference path. Repo on GitHub, artifacts on Hugging Face.
Prefill (C=1 to C=8)
- Qwen3.6-35B-A3B nvfp4: 5,838 tok/s single-lane → 21,897 at 8 concurrent
- Qwen3.8-27B nvfp4: 4,034 → 8,786 tok/s
- Gemma 4 26B-A4B (QAT int4): 2,729 → 6,626 tok/s
Decode (plain vs MTP)
- Qwen3.6-35B-A3B nvfp4: 2,142 tok/s at 8 lanes with MTP vs 1,338 plain
- Qwen3.8-27B nvfp4: 1,047 with MTP vs 545 plain
- Gemma 4 26B-A4B: 500 tok/s single-lane MTP at 0.87-0.95 acceptance
- Gemma 4 31B can't fit 8 MTP lanes in 32GB; unmeasured cells are documented
The author is explicit about methodology: single-lane figures are true per-request rates, multi-lane are point aggregates including decode time, and decode is measured over steady one-second windows with adaptive draft lengths noted.
More from Infra
- SemiAnalysis teases TPUv7 TCO data: $0.181/M tokens vs $0.222 B200 — firstadopter · 2026-10-14
- Serverless tool calling could slash agent costs 30x by running 1000 agents per sandbox — madhavsinghal_ · 2026-10-14
- Prompt caching kills easy/hard model routing: cached big model now cheaper than small one — Lance_Saul_85 · 2026-10-14
- BofA lifts global memory TAM forecast to $2T by 2030, run rate already over $1T — Beth_Kindig · 2026-10-14
- Ayar Labs, Wiwynn and GUC demo integrated optical compute system with live telemetry at OCP — bookwormengr · 2026-10-14
- AnythingLLM ditches Ollama for Lemonade, shrinking engine from 1.8GB to 137MB — jfowers_amd · 2026-10-14