QuixiInfer: Rust Engine Runs 125B MoE Model on a Single RTX 5090
QuixiAI · x · 2026-11-20
QuixiAI released QuixiInfer, an open-source inference engine written from scratch in Rust that serves Qwen3.8-Flash-Next (125B MoE, 512 experts per layer, 262k context) from a single workstation: one RTX 5090 (32GB), a 16-core CPU, 128GB RAM and an NVMe SSD.
- Ships its own CUDA kernels launched via CubeCL; no ggml or llama.cpp code is linked
- Server design scaled down from DeepSeek's cluster engine DeepInfer, with CPU expert offloading
- Features: continuous batching, async scheduling, paged FP8 KV, shared prefix cache, exact speculative decoding, structured outputs
- Benchmarked at 25-30 tok/s with up to 3 concurrent requests on a 5090
- Uses dynamic CPU and NVMe offloading; author is recruiting testers on 3090/4090 and other GPUs
More from Infra
- Microsoft's Bing Web IQ Goes GA on Azure: Auto Orchestration Delivers Fresh Grounding for Agents — dejanseo · 2026-11-20
- exe.dev ships Rex integration, making remote VMs the default agentic dev workstation — davidcrawshaw · 2026-11-20
- Ex-Waymo exec Abe Murray: CUDA and ISAs won't stay moats, huge chip innovation incoming — IgorCarron · 2026-11-20
- MIT and NVIDIA's QDA uses agents to auto-design LLM quantization recipes, beating top public ones by up to 2.2 points — omarsar0 · 2026-11-20
- Caixin: US remote-access cutoff and Nvidia CEO exit leave China's tech giants halting overseas compute deals — Gregory_C_Allen · 2026-11-20
- PostgreSQL 19 Ships WAIT FOR LSN, Solving Read-Your-Writes on Replicas — Simon Willison · 2026-11-20