NInfer Enables Efficient Qwen3.8-27B Inference on RTX 3090
mrmontanasagrada · reddit · 2026-08-15
A developer ported the NInfer runtime to RTX 3090 (24GB) to optimize Qwen3.8-27B inference. Benchmarks show the model achieves 71 tokens/s for single requests and 165 tokens/s decoding across 8 concurrent requests, utilizing ReplaySSM and CUDA Graphs while staying within 24GB VRAM. The project demonstrates the potential of running large models on older consumer GPUs, with preliminary results also mentioned for the Qwen3-35B-A3B model.
Related event: RTX 3090 Optimized for Efficient Qwen3.8-27B Inference(3 posts)→
More from Infra
- Phison CEO calls CXMT memory scare a 'ghost story', warns of materials shortage — BenBajarin · 2026-10-16
- Ben Bajarin: The Future of Compute Is Fungible as Hyperscalers Design for Workload Flexibility — BenBajarin · 2026-10-16
- AirLLM runs frontier-size models layer-by-layer: 671B on 12GB VRAM — Aiden_Tech_Ai · 2026-10-16
- NVIDIA Launches cuPotential in CUDA-X to Bring AI Agents to Semiconductor Materials Simulation — tomaszbednarz · 2026-10-16
- Q4-quantized coder model on 8-core CPU builds interactive solar system demo in one shot — IntrepidMindExplorer · 2026-10-16
- TokenSpeed MLA 2 ships for Vera Rubin: up to 3.44× faster Kimi K3 decode — zhyncs42 · 2026-10-16