Perplexity details exabyte-scale embedding serving stack: up to 4.8x faster p99 than vLLM
inductionheads · x · 2026-09-05
Perplexity published research on the SoTA serving infrastructure behind its pplx-embed models and rerankers, which back an exabyte-scale search index.
Key insight: embedding models have the same serving profile as LLMs — batch indexing is compute-bound prefill, online serving is memory-bound decode — so they reuse the optimized CUDA kernels built for large LLMs and get high efficiency for free.
Runtime optimizations:
- Whole-model CUDA graphs captured lazily as the engine serves
- A Rust LazyTensor overlapping CPU scheduling with GPU execution
Results: on BGE-M3 at 128 tokens with a single H200, up to 3x lower p50 and 4.8x lower p99 latency than vLLM.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Nvidia guides 70% growth citing unmet AI demand, while media panics over a $500M Korean neocloud IPO — firstadopter · 2026-10-21
- Pichai: everyone is massively short on AI compute — 'everyone wishes you had two times what you had planned' — firstadopter · 2026-10-21
- Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x — stanfordnlp · 2026-10-21
- General Compute partners with Positron AI: Atlas memory-centric hardware ships Q1 2027 — aftahi_ai · 2026-10-21
- AI bond issuance hits $419B in 2026, up 90% from all of 2025 — SumitGup · 2026-10-21