Perplexity's Tulip: CUDA graphs and LazyTensors for low-latency batch embedding
perplexity_ai · x · 2026-09-05
Perplexity explains Tulip, its lightweight Rust gRPC inference server between Ivy and the ROSE engine: it batches incoming requests for the GPU, noting that for small embedding models runtime depends on tokens (512 tokens fills the GPU). Two techniques cut latency and boost throughput: CUDA graphs pre-record GPU work for single-call launches, and LazyTensors tracks results asynchronously so the CPU preps the next batch while the GPU finishes the current one.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- acs infra launches free base model API offering Llama 3.1 8B and 405B for LLM psychology researchers — amplifiedamp · 2026-10-21
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Nvidia guides 70% growth citing unmet AI demand, while media panics over a $500M Korean neocloud IPO — firstadopter · 2026-10-21
- Pichai: everyone is massively short on AI compute — 'everyone wishes you had two times what you had planned' — firstadopter · 2026-10-21
- Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x — stanfordnlp · 2026-10-21
- General Compute partners with Positron AI: Atlas memory-centric hardware ships Q1 2027 — aftahi_ai · 2026-10-21