Running Qwen 1M context on dual 5090s: custom engine beats vLLM at decode
Littlepharaoh · reddit · 2026-08-30
A developer forked the NInfer engine to add tensor parallelism and YaRN scaling, enabling a 1M token context window for Qwen on dual RTX 5090s (27.4 GB per card, no NVLink).
Performance highlights (500W per GPU):
- Decode: At 653k context, NInfer hits 119 tok/s with MTP speculative decoding vs 57 without, while vLLM achieves only 42 tok/s.
- Long Context: Beyond vLLM's native 262k limit, MTP acceptance drops to 0%. NInfer maintains 55–60% acceptance out to 1M tokens.
- Prefill: vLLM is faster (1.2–1.3x), taking 18 minutes for a full 1M prompt fill on NInfer.
- Memory: vLLM's fp8 KV cache caps at 759k tokens; NInfer's INT8 KV fits the full 1,048,576 tokens.
More from Infra
- Chip equipment beats bull case: 2026 WFE seen at $153B, $278B by 2030 — BenBajarin · 2026-10-16
- Steam now launches in 5.23s, beating Epic's 5.47s — and vLLM gets roasted for slow startup — AlpinDale · 2026-10-16
- Shadeform to Announce Six New Features on Oct 19: Lower GPU Costs, More Suppliers — _jaydeepkarale · 2026-10-16
- Fastino launches Training API for SFT/RL, claiming 25-85% cost cut vs Tinker — kalyan_kpl · 2026-10-16
- Dallas Fed: AI data center debt issuance equals ~1/8 of Treasury duration supply — _onionesque · 2026-10-16
- Running Wan2.1 14B FP16 on a 12GB laptop GPU: VRAM swap stress test finishes in 4m11s — RYuu-1120 · 2026-10-16