Perplexity's three-layer serving stack: Ivy HTTP, Tulip gRPC, ROSE engine
perplexity_ai · x · 2026-09-05
Perplexity summarizes its standardized inference API stack in three layers: Ivy (Rust HTTP) handles parsing, tokenization, and templating on the CPU and translates to gRPC; Tulip (Rust gRPC) schedules batches for the ROSE engine; ROSE (Python) implements forward passes and CUDA graph management.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- Corporate token spending growth collapses from 42% to 0.2% as hyperscaler ROIC falls to 20% — SumitGup · 2026-10-21
- Qualcomm plans worldwide performance-based layoffs as it chases $5B AI data center revenue — rohanpaul_ai · 2026-10-21
- vLLM at PyTorchCon 2026: KV cache and kernel optimizations across TPU, AMD, and NVIDIA — PyTorch · 2026-10-21
- antirez to port DeepSeek Infer kernels to DGX Spark's SM121 for DwarfStar — antirez · 2026-10-21
- Ternary-Bonsai-2-9B: 2-bit ternary vision-language model trends on Hugging Face for on-device use — prism-ml · 2026-10-21
- General Compute x Positron: GPUs Handle Prefill, Memory-First Chips Tackle Decode-Bound LLM Inference — cneuralnetwork · 2026-10-21