Open-source PolyServe autotuner boosts LLM throughput 56–97% across three GPUs
AffectionateSir8341 · reddit · 2026-09-17
After seeing endless 'best config' questions on the vLLM Slack, a developer built PolyServe, an open-source LLM autotuner that benchmarks configurations across vLLM, SGLang, and llama.cpp, picks one under your latency constraints, and serves it behind an OpenAI-compatible API.
Measured vs a fixed tuning heuristic (Llama 3.1 8B):
- RTX 4090: 667 → 1,041 tok/s (+56%)
- A100: 674 → 1,155 tok/s (+71%)
- A40: 257 → 506 tok/s (+97%)
- Baseline uses fp8 weights, 8-bit KV cache, and batch target 256; evaluated on held-out prompts with three interleaved runs per row.
Vs stock defaults (Qwen2.5-3B, A40): PolyServe hits 619 tok/s — 23% over default vLLM, 30% over default SGLang — with p95 TTFT of 157 ms within a 500 ms budget.
Key trade-off: Qwen2.5-14B on a 24 GB RTX 4090 with 4-bit allowed jumps from 365 to 1,412 tok/s and cuts energy per token by 65%, at GSM8K 186/200 vs 190/200, so 4-bit stays opt-in.
Usage: pip install polyserve, then polyserve serve <model> --workload chat. First launch calibrates (22–56 min), later launches reuse cached results. MIT-licensed, repo includes raw benchmarks and quality checks.
More from Infra
- Speculative decoding: small draft model proposes tokens, big model verifies in one pass — HowDevelop · 2026-09-17
- India to invest roughly $30 billion in building a domestic semiconductor industry — Polymarket · 2026-09-17
- Dev Inspired by AgentConf Keynote to Run All Agents Locally on Home Blackwell GPU Stack — TejasKumar_ · 2026-09-17
- DeepSeek-V4.1 Flash deep dive: pushing KV cache compression to the limit at 420 tok/s — teortaxesTex · 2026-09-17
- Four Scheduling Techniques Flatten MoE Training Memory Peaks, Enabling 1M Context at 10.4x Throughput — Shrey Pandit · 2026-09-17
- GLM agent built its own inference infra in two weeks, tripling end-to-end throughput — jietang · 2026-09-17