456 tok/s Qwen 3.8 on Modded RTX 2080 Ti via NInfer Port
xrailgun · reddit · 2026-08-21
The author ported and tuned the NInfer C++/CUDA inference engine, originally designed for RTX 5090/Blackwell, to NVIDIA Turing architecture (sm75), specifically targeting a modded 22GB RTX 2080 Ti.
Benchmarks (Qwen 3.8-27B MTP3, W8A16, Q8 KV):
- Standard Autoregressive (MTP0): 25 tok/s.
- Speculative Decoding (MTP3, draft window=3): 456 tok/s (65% acceptance rate).
- VRAM Usage: 17.5 GiB total, leaving 4.5–5.0 GiB free for KV cache, enabling 128k context.
The project includes OpenAI/Anthropic-compatible HTTP serving (streaming, function calling) and is open-sourced on GitHub.
More from Infra
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16
- Meta launches Muse Spark Open, says it runs on two M5 Ultra Mac Studios with Apple's help — The Verge AI · 2026-10-16
- vLLM ships Day-0 LoRA RL for Muse Spark Open, lifting AIME from 0.56 to 0.85 — vllm_project · 2026-10-16
- DSpark draft model for Muse Spark Open proposes 7 tokens per pass with 23-42% acceptance — vllm_project · 2026-10-16