Rust Inference Engine Paddock Benchmarks: 3.5x Faster TTFT Than vLLM on Qwen
saltexx · reddit · 2026-08-22
A new inference engine written from scratch in Rust, Paddock, has been released with benchmarks showing significant performance gains over vLLM and llama.cpp. In a Qwen3.6-27B benchmark with 32 concurrent clients, Paddock achieved a TTFT of 697ms, compared to 2.5s for vLLM and 6.9s for llama.cpp. It won 11 of 13 scenarios against vLLM and all 13 against llama.cpp.
Key Features & Architecture:
- Built in Rust with Custom CUDA Kernels: Designed for "one model per GPU" to maximize memory bandwidth without tensor parallelism, minimizing failure blast radius.
- Zero-Config Setup: Unpack and run paddock to launch the built-in Studio, which fetches models and starts serving. Offers OpenAI/Anthropic-compatible endpoints, plus embeddings, reranking, and STT.
- Speculative Decoding Gains: With MTP and DFlash draft designs enabled on an RTX PRO 6000, single-stream throughput jumped from 46.8 to 202 tok/s, and 32-concurrent throughput from 1005 to 1285 tok/s.
Caveats:
- Hardware Targeting: Fast paths are optimized for Blackwell (e.g., 5090/RTX PRO 6000); Ampere is the supported floor, with reduced gains on older cards.
- Licensing: Free to use (no usage limits) for individuals and companies, but the source code is not open source.
More from Infra
- MCP Isn't Replacing APIs: It's Changing Who APIs Are Designed For — kush_patil · 2026-08-22
- Data Center Opposition Surged from 42 to 75 Percent in One Year — The Decoder · 2026-08-22
- Qwen3.8-27B gets DFlash2 speculative-decoding GGUF release for llama.cpp — incoai · 2026-08-22
- Woolly post-trains Qwen3-8B for 2–3× faster math & code decoding — bosmeny · 2026-08-22
- Opinion: States Banning Data Centers Face 20 Years of Economic Depression — GabGarrett · 2026-08-22
- Vercel Fixes TLS Fragmentation Issue, All Websites Back Online — uwukko · 2026-08-22