Perplexity Reveals Its In-House Embedding Inference Stack
On September 5, Perplexity's official account published a series of tweets and a research article detailing the in-house SoTA embedding and ranking model inference infrastructure underpinning its search answer pipeline, claiming lower latency and higher throughput for both online and batch workloads, delivering faster search and lower costs than off-the-shelf solutions.
Confirmed
- Retrieval foundation: queries and documents are embedded into the same vector space and searched by nearest neighbors, creating two workload types—throughput-oriented batch embedding (for indexing and scoring) and latency-oriented single-query online embedding (for real-time search)—each requiring separate optimization.
- Three-layer architecture: Ivy (a Rust HTTP gateway) handles CPU-side request preparation—parsing, tokenization, and templating—and splits large batch requests before sending them via gRPC to Tulip, allowing request formats and tokenization logic to be tuned independently without touching inference servers; Tulip (a Rust gRPC inference server) sits between Ivy and ROSE, collecting requests, batching them for the GPU, and using CUDA graphs and LazyTensors to cut latency and boost throughput; ROSE (a Python engine) lets LLMs and embedding models reuse the same set of kernels, skipping the KV cache during embedding inference, using ragged attention instead of paged attention, and supporting multiple attention backends.
- Performance characteristics: for small embedding models, runtime depends on token count rather than query count, with about 512 tokens enough to saturate the GPU.
Why It Matters
- Perplexity migrated the embedding retrieval pipeline at the heart of its search engine from off-the-shelf inference solutions to an in-house stack, publishing the full layered design and optimization details—a rare reference sample for the engineering practice of LLM retrieval systems.
2026-09-05 ~ 2026-09-05 · 8 related posts
Primary sources
- Perplexity details SoTA embedding serving stack: Ivy gateway, Tulip server, ROSE engine — perplexity_ai ·
- Perplexity claims faster, cheaper search with in-house Ivy/Tulip/ROSE stack — perplexity_ai ·
- Perplexity details exabyte-scale embedding serving stack: up to 4.8x faster p99 than vLLM — inductionheads ·
- [source] Perplexity details SoTA embedding serving stack: Ivy gateway, Tulip server, ROSE engine — perplexity_ai · 2026-09-05
- Perplexity explains its embedding search: batch vs online workloads in one vector space — perplexity_ai · 2026-09-05
- Perplexity's three-layer serving stack: Ivy HTTP, Tulip gRPC, ROSE engine — perplexity_ai · 2026-09-05
- Perplexity's Tulip: CUDA graphs and LazyTensors for low-latency batch embedding — perplexity_ai · 2026-09-05
- Perplexity's ROSE engine: ragged attention, no KV cache for embeddings — perplexity_ai · 2026-09-05
- Perplexity's Ivy: the HTTP gateway handling tokenization and batch splitting — perplexity_ai · 2026-09-05
- [source] Perplexity claims faster, cheaper search with in-house Ivy/Tulip/ROSE stack — perplexity_ai · 2026-09-05
- [source] Perplexity details exabyte-scale embedding serving stack: up to 4.8x faster p99 than vLLM — inductionheads · 2026-09-05