Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack

On September 5, Perplexity published a series of posts and a research article detailing the inference infrastructure behind its search pipeline, which powers the pplx-embed embedding and reranker model families and an EB-scale search index. The company says both online and batch workloads achieve lower latency and higher throughput than off-the-shelf solutions, enabling faster search at lower cost; @inductionheads, resharing the blog, reported the embedding service is 3-4.8x faster than vLLM.

Confirmed

• Retrieval foundation: queries and documents are embedded into the same vector space and searched via nearest neighbors, creating two workloads—throughput-oriented batch embedding (indexing and scoring) and latency-oriented single-query online embedding (real-time search), each optimized separately.
• Three-layer architecture: Ivy (Rust HTTP gateway) handles CPU-side request preparation—parsing, tokenization, templating—and splits large batches before forwarding via gRPC, letting request formats and tokenization be tuned without touching inference servers; Tulip (Rust gRPC inference server) sits between Ivy and ROSE, collecting requests, batching them for the GPU, and using CUDA graphs and LazyTensors to cut latency and boost throughput; ROSE (Python engine) lets LLMs and embedding models share the same kernels, skipping KV cache during embedding inference by replacing paged attention with ragged attention, with support for multiple attention backends.
• Performance characteristics: for small embedding models, runtime depends on token count rather than query count, with roughly 512 tokens enough to saturate the GPU.

Why it matters

• Perplexity migrated the core embedding retrieval path of its search engine from off-the-shelf inference stacks to a self-built one and published the full layered design and optimization details—a rare engineering reference for large-model retrieval systems.

2026-09-05 ~ 2026-09-05 · 8 related posts

• [source] Perplexity details SoTA embedding serving stack: Ivy gateway, Tulip server, ROSE engine — perplexity_ai · 2026-09-05
• Perplexity explains its embedding search: batch vs online workloads in one vector space — perplexity_ai · 2026-09-05
• Perplexity's three-layer serving stack: Ivy HTTP, Tulip gRPC, ROSE engine — perplexity_ai · 2026-09-05
• Perplexity's Tulip: CUDA graphs and LazyTensors for low-latency batch embedding — perplexity_ai · 2026-09-05
• Perplexity's ROSE engine: ragged attention, no KV cache for embeddings — perplexity_ai · 2026-09-05
• Perplexity's Ivy: the HTTP gateway handling tokenization and batch splitting — perplexity_ai · 2026-09-05
• [source] Perplexity claims faster, cheaper search with in-house Ivy/Tulip/ROSE stack — perplexity_ai · 2026-09-05
• Perplexity details exabyte-scale embedding serving stack: up to 4.8x faster p99 than vLLM — inductionheads · 2026-09-05