QuixiInfer: Rust Inference Engine Runs DeepSeek with 256k Context on a Single 5090 at 25-30 tok/s
hotschmoe · x · 2026-11-20
Developer QuixiAI released QuixiInfer, a from-scratch Rust inference engine that runs DeepSeek-v4.1-Flash-1017 with 256k context on a single GPU.
It hits 25-30 tok/s with up to 3 concurrent requests on an RTX 5090, using dynamic CPU and NVMe offloading for weights that don't fit in VRAM. Inspired by DeepInfer and Strata. The author is seeking contributors to test on other GPUs (3090, 4090...), and also maintains QuixiCore-xpu with optimized B70 kernels.
Related event: Rust-Built QuixiInfer Runs DeepSeek MoE on a Single RTX 5090(3 posts)→