NInfer Adds Day-0 Support for Qwen3.8-27B, Hits ~200 tok/s on RTX 5090
FormOne2615 · reddit · 2026-08-15
NInfer inference engine now supports Qwen3.8-27B on day zero, achieving 200 tok/s generation on a single RTX 5090 with speculative decoding.
Key improvements include:
- Up to 8 concurrent requests with a shared paged KV cache pool, each retaining full context length.
- ReplaySSM for GDN + speculative decoding, greatly reducing recurrent-state memory overhead under concurrency (not fully supported by vLLM yet).
- Numerous CUDA kernel optimizations and PDL usage to further reduce latency.
Weights are available on Hugging Face; feedback and bug reports are welcome.
More from Infra
- Three CoreWeave co-founders have sold $3.3B in shares since IPO amid credit worries — SumitGup · 2026-10-16
- Cherry pi on a Raspberry Pi: the tiny-model rumor developers are watching — sull · 2026-10-16
- CoreWeave's debt costs falling: DDTL spreads tightened from S+9.62% to S+2.25% — BenBajarin · 2026-10-16
- SoftBank lines up US IPOs for SB Energy and AI robotics spinoff Roze, valuations top $50B — nordicinst · 2026-10-16
- TRL reproduces Modal's mixed-precision async RL: plain GRPO collapses, SC+MIS holds 0.72 — QGallouedec · 2026-10-16
- Hugging Face scales Indic pretraining with 2.8T tokens and an open data pipeline — Hugging Face Blog · 2026-10-16