Perplexity details exabyte-scale embedding serving stack: up to 4.8x faster p99 than vLLM
inductionheads · x · 2026-09-05
Perplexity published research on the SoTA serving infrastructure behind its pplx-embed models and rerankers, which back an exabyte-scale search index.
Key insight: embedding models have the same serving profile as LLMs — batch indexing is compute-bound prefill, online serving is memory-bound decode — so they reuse the optimized CUDA kernels built for large LLMs and get high efficiency for free.
Runtime optimizations:
- Whole-model CUDA graphs captured lazily as the engine serves
- A Rust LazyTensor overlapping CPU scheduling with GPU execution
Results: on BGE-M3 at 128 tokens with a single H200, up to 3x lower p50 and 4.8x lower p99 latency than vLLM.
Related event: Perplexity Reveals Its In-House Embedding Inference Stack(8 posts)→
More from Infra
- Burn Bar for Omarchy visualizes Claude/Codex token burn, quotas and GPU load locally — DanWahlin · 2026-09-05
- Scaling wall? Reddit argues test-time compute is the industry's new playbook — erdematar · 2026-09-05
- Tencent Hunyuan Hy4 preview: 770B total/49B active, 1M context, Apache 2.0, day-0 vLLM — aftahi_ai · 2026-09-05
- Qwen3.8 27B Quant Fits 24GB VRAM at 100k Context, Sparking Local Model Profit-Threat Debate — ChopSticksPlease · 2026-09-05
- Speechify CEO on self-built data centers, ElevenLabs leapfrog, and the $15M AI talent war — 20VC · 2026-09-05
- He Uses Local LLMs Like a 3D Printer: 12 Games, 29 Mods and Countless Tools Built Solo — Quebber · 2026-09-05