Perplexity's Tulip: CUDA graphs and LazyTensors for low-latency batch embedding
perplexity_ai · x · 2026-09-05
Perplexity explains Tulip, its lightweight Rust gRPC inference server between Ivy and the ROSE engine: it batches incoming requests for the GPU, noting that for small embedding models runtime depends on tokens (512 tokens fills the GPU). Two techniques cut latency and boost throughput: CUDA graphs pre-record GPU work for single-call launches, and LazyTensors tracks results asynchronously so the CPU preps the next batch while the GPU finishes the current one.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- Sizing a 1 GW datacenter: ~522,000 Blackwell GPUs fit inside — bookwormengr · 2026-10-21
- DeepSeek's production density reportedly hits ~$38 per GPU-hour at saturation — bookwormengr · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21
- DeepSeek open-sources DeepInfer, the vLLM-based engine behind V4.1-Flash — The Decoder · 2026-10-21
- Robotics startup launches Deployments to serve fine-tuned HF checkpoints to robots in real time — arnie_hacker · 2026-10-21