Perplexity details SoTA embedding serving stack: Ivy gateway, Tulip server, ROSE engine
perplexity_ai · x · 2026-09-05
Perplexity published research on the SoTA serving infrastructure behind the embedding and ranking models that power every search answer.
The official thread breaks down the architecture:
- Ivy: a Rust HTTP gateway handling CPU-side request prep (parsing, tokenization, templating) before forwarding over gRPC;
- Tulip: a lightweight Rust gRPC inference server that batches requests, using CUDA graphs to cut CPU launch overhead and LazyTensors for async pipelining;
- ROSE: the Python model engine sharing kernels between LLMs and embeddings, skipping KV cache for embeddings with ragged attention, and supporting multiple attention backends chosen by model shape and sequence length.
The stack targets two workloads—throughput-focused batch embedding and latency-focused online embedding—and Perplexity claims faster search at lower cost than off-the-shelf solutions.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Decision Models' routing classifier hits AntSeed, cutting inference cost to 24% of always-Opus — airesearch12 · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20
- a16z's Ben Horowitz says AI buildout isn't the 90s bubble — an investor pushes back — henloitsjoyce · 2026-10-20
- Micron and Nvidia drive ~36% of S&P 500's Q3 earnings growth — Beth_Kindig · 2026-10-20
- NVIDIA lands in Europe: Jensen Huang and German chancellor discuss industrial AI at GTC Berlin — nvidia · 2026-10-20