Paddock Open-Sourced: Rust/C++ Inference Engine With Own CUDA Kernels Beats vLLM in All 13 Cells
saltexx · reddit · 2026-09-04
Truespar kept its September promise and open-sourced Paddock under MIT/Apache-2.0 — the repo is their internal one, custom kernels included.
- Rust + C++ inference engine with in-house CUDA kernels, one binary exposing OpenAI/Anthropic-style APIs, loads GGUF and safetensors; runs 300B tokens/year in production
- Benchmarks (Qwen3.8-27B FP8, single RTX PRO 6000, spec decoding off): faster than vLLM in 13/13 cells (1.02x–1.19x), faster than SGLang in 10/13, and 1.5x–37x over llama.cpp Q80; 32 clients at 1024 in/1024 out hits 1062 tok/s vs vLLM's 958 and SGLang's 844
- Limitations: CUDA only (Windows/Linux), validated on Blackwell and Ampere (Ada kernels unvalidated behind an env flag), no Mac/ROCm/Vulkan, one model per GPU, no tensor parallelism
More from Infra
- Microsoft names Project Zenith: dev-focused Windows shipping with AMD Ryzen AI Halo chips — tomwarren · 2026-09-04
- Reframe render engine claims ~48x speedup over Octane and Arnold after months of kernel rewrites — D3VAUX · 2026-09-04
- Local Qwen3 8B Flash on RTX 6000 Pro: 2000 tps prefill but only 40 tps decode — AppealSame4367 · 2026-09-04
- Voz hits 5-20x faster on-device speech-to-text by heavily optimizing for Apple's ANE — pcuenq · 2026-09-04
- Gemini 3.8 Flash: same token price, different cost per task — a real-world billing test — JuggernautCritical92 · 2026-09-04
- DIT.ai launches AI token exchange routing GPT, Claude, Gemini at 30-70% below list price — SucceededMind · 2026-09-04