Perplexity's Ivy: the HTTP gateway handling tokenization and batch splitting
perplexity_ai · x · 2026-09-05
Perplexity details Ivy, the HTTP gateway in its serving stack: it handles CPU-side request prep (parsing, tokenization, templating) and splits large batches before sending to Tulip over gRPC, letting the team tune request formatting and tokenization without touching the heavier inference servers.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- Sizing a 1 GW datacenter: ~522,000 Blackwell GPUs fit inside — bookwormengr · 2026-10-21
- DeepSeek's production density reportedly hits ~$38 per GPU-hour at saturation — bookwormengr · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21
- DeepSeek open-sources DeepInfer, the vLLM-based engine behind V4.1-Flash — The Decoder · 2026-10-21
- Robotics startup launches Deployments to serve fine-tuned HF checkpoints to robots in real time — arnie_hacker · 2026-10-21