NInfer hits 542 tok/s on Qwen3.6-35B-A3B on a single RTX 5090
FormOne2615 · reddit · 2026-07-20
The author open-sourced NInfer, a from-scratch C++/CUDA inference engine optimized for two specific Qwen3.6 checkpoints on a single RTX 5090.
Main result: Qwen3.6-35B-A3B sustained 542.8 tok/s over a full 65,536-token completion on one GPU. The post includes benchmark tables for long reasoning, code, translation, story, and structured-output workloads, plus context-length scaling data up to 260k tokens. It also reports AIME25/AIME26/GPQA-Diamond scores for the converted artifacts.
NInfer is not a general-purpose engine; it uses custom quantization, weight layouts, kernel fusion, and a dedicated LM-head draft path. The repo and converted model artifacts are publicly available, with support limited to the two listed Qwen3.6 models and primarily RTX 5090-class hardware.
More from Infra
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11
- PyTorch Day Korea 2026 launches first offline conf, CFP closes Sept 13 — PyTorch · 2026-09-11
- Local LLM server dilemma: 4x CMP-170HX (price up 53% in 20 days) vs Mac Studio M5 Ultra — rumboll · 2026-09-11
- llama.cpp lands Flash Attention tuning for RDNA4, big prefill gains on AMD — pmttyji · 2026-09-11