QuixiInfer: from-scratch Rust engine runs DeepSeek with 256k context on a single RTX 5090 at ~30 tok/s
QuixiAI · x · 2026-11-20
Developer QuixiAI released QuixiInfer, a from-scratch Rust inference engine that runs DeepSeek-v4.1-Flash-1017 on a single RTX 5090 with 256k context and up to 3 concurrent requests at 25-30 tok/s, using dynamic CPU and NVMe offloading (inspired by DeepInfer and Strata).
Benchmarks vs Strata:
- Prefill: 1.6x faster at 16k tokens (1,392 vs 854 tok/s), up to 3.4x at 2k (2.3s vs 7.3s)
- Decode: 1.5x faster right after a prompt; steady state 32.4-32.7 tok/s vs Strata's 27.2
The author is seeking contributors to test on 3090, 4090, and AMD GPUs.
Related event: Rust-Built QuixiInfer Runs DeepSeek MoE on a Single RTX 5090(3 posts)→
More from Infra
- Nvidia down ~40% from October peak as Anthropic IPO pops 34% above expectations — The Verge AI · 2026-11-20
- Ion Stoica explains why complex AI workloads increasingly fit Ray's design — PyTorch · 2026-11-20
- GPU rents up, token prices down: inference providers' break-even load jumps 22%–75% in 5 months — AccBalanced · 2026-11-20
- Missing config file was throttling GPUDirect Storage — fix yields 3x reads, 9x writes — AccBalanced · 2026-11-20
- NVIDIA Dynamo cuts RL rollout bottlenecks with KV-aware routing and delta weight updates — NVIDIAAI · 2026-11-20
- Arm reaches the AI laptop: Surface Laptop Ultra runs 120B models locally — kimmonismus · 2026-11-20