Paddock Open-Sourced: Rust/C++ Inference Engine With Own CUDA Kernels Beats vLLM in All 13 Cells
saltexx · reddit · 2026-09-04
Truespar kept its September promise and open-sourced Paddock under MIT/Apache-2.0 — the repo is their internal one, custom kernels included.
- Rust + C++ inference engine with in-house CUDA kernels, one binary exposing OpenAI/Anthropic-style APIs, loads GGUF and safetensors; runs 300B tokens/year in production
- Benchmarks (Qwen3.8-27B FP8, single RTX PRO 6000, spec decoding off): faster than vLLM in 13/13 cells (1.02x–1.19x), faster than SGLang in 10/13, and 1.5x–37x over llama.cpp Q80; 32 clients at 1024 in/1024 out hits 1062 tok/s vs vLLM's 958 and SGLang's 844
- Limitations: CUDA only (Windows/Linux), validated on Blackwell and Ampere (Ada kernels unvalidated behind an env flag), no Mac/ROCm/Vulkan, one model per GPU, no tensor parallelism
More from Infra
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Decision Models' routing classifier hits AntSeed, cutting inference cost to 24% of always-Opus — airesearch12 · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20
- a16z's Ben Horowitz says AI buildout isn't the 90s bubble — an investor pushes back — henloitsjoyce · 2026-10-20
- Micron and Nvidia drive ~36% of S&P 500's Q3 earnings growth — Beth_Kindig · 2026-10-20
- NVIDIA lands in Europe: Jensen Huang and German chancellor discuss industrial AI at GTC Berlin — nvidia · 2026-10-20