Single RTX 5090 serves 125B MoE at 256k context: QuixiInfer hits 27 tok/s decode
QuixiAI · x · 2026-11-08
QuixiAI open-sourced QuixiInfer, a Rust inference engine with custom CUDA kernels (compiled via CubeCL, no ggml/llama.cpp) that serves Qwen3.8-Flash-Next — a 125B MoE with 512 experts per layer and 262k context — to multiple users from a single RTX 5090 (32GB), 16-core CPU, and 128GB RAM.
Key details:
- Server design derived from DeepSeek's cluster engine DeepInfer, scaled down to one GPU with CPU expert offload
- Continuous batching, async scheduling, paged FP8 KV cache, shared prefix cache, exact speculative decoding, structured outputs
- Runs NVIDIA's NVFP4 checkpoint: routed experts at 4-bit, attention and shared experts at higher precision
Author benchmarks: 27 tok/s decode and 161 tok/s prefill at 256k context.
Related event: Running DeepSeek V4.1 Flash Locally: 27 tok/s on an RTX 5090(3 posts)→
More from Infra
- TSMC denies 3D DRAM plans at Nvidia's request, stays pure-play logic foundry — rohanpaul_ai · 2026-11-08
- TSMC denies developing 3D DRAM at Nvidia's request, staying a pure logic foundry — rohanpaul_ai · 2026-11-08
- Why is everyone sleeping on Modal live inference endpoints for agents? — vboykis · 2026-11-08
- Arista CEO forecasts 10+ GW of AI deployments across 10+ accelerator types by end of 2027 — Beth_Kindig · 2026-11-08
- Hugging Face launches live training streams, starting with RL-tuned LoRA adapters — art_zucker · 2026-11-08
- Lobsters thread: how AI traffic is degrading the web for human developers — vboykis · 2026-11-08