Perplexity details SoTA embedding serving stack: Ivy gateway, Tulip server, ROSE engine
perplexity_ai · x · 2026-09-05
Perplexity published research on the SoTA serving infrastructure behind the embedding and ranking models that power every search answer.
The official thread breaks down the architecture:
- Ivy: a Rust HTTP gateway handling CPU-side request prep (parsing, tokenization, templating) before forwarding over gRPC;
- Tulip: a lightweight Rust gRPC inference server that batches requests, using CUDA graphs to cut CPU launch overhead and LazyTensors for async pipelining;
- ROSE: the Python model engine sharing kernels between LLMs and embeddings, skipping KV cache for embeddings with ragged attention, and supporting multiple attention backends chosen by model shape and sequence length.
The stack targets two workloads—throughput-focused batch embedding and latency-focused online embedding—and Perplexity claims faster search at lower cost than off-the-shelf solutions.
Related event: Perplexity Reveals Its In-House Embedding Inference Stack(8 posts)→
More from Infra
- Will combining multiple GPUs' VRAM for local LLMs ever work out of the box? — PusheenHater · 2026-09-05
- Declarative Attention lets LLMs declare their own focus, cutting 52% of KV cache reads — eigenlaplace · 2026-09-05
- Agent outputs die when the VM sleeps: octomind's design for deliverables that survive — donk8r · 2026-09-05
- Japan to develop AI-powered satellites — AIFlow_ML · 2026-09-05
- Hybrid Compute on Mac ships with open-sourced local inference engine and PII classifier — andrewgwils · 2026-09-05
- Tesla's RIM process kills the paint shop, shrinking Cybercab factory footprint ~50% — elonmusk · 2026-09-05