Perplexity's Ivy: the HTTP gateway handling tokenization and batch splitting
perplexity_ai · x · 2026-09-05
Perplexity details Ivy, the HTTP gateway in its serving stack: it handles CPU-side request prep (parsing, tokenization, templating) and splits large batches before sending to Tulip over gRPC, letting the team tune request formatting and tokenization without touching the heavier inference servers.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- Developer Moves AI Agents to a Dedicated Linux Workstation Using Incus System Containers — kevinkern · 2026-10-21
- Nvidia guides 70% growth citing unmet AI demand, while media panics over a $500M Korean neocloud IPO — firstadopter · 2026-10-21
- Pichai: everyone is massively short on AI compute — 'everyone wishes you had two times what you had planned' — firstadopter · 2026-10-21
- Stanford CS336 Lecture 6: fused Triton kernels cut softmax memory traffic ~4x — stanfordnlp · 2026-10-21
- General Compute partners with Positron AI: Atlas memory-centric hardware ships Q1 2027 — aftahi_ai · 2026-10-21
- AI bond issuance hits $419B in 2026, up 90% from all of 2025 — SumitGup · 2026-10-21