Modal Engineers Detail Sticky Serverless Routing for Real-Time AI Inference
blelbach · x · 2026-10-21
Modal engineers Daniel Shaar, Siwon Park, Kartik Sinha, and Kathir Meyyappan published "Serverless routing made sticky", explaining how Modal's serverless compute — which powers real-time inference, training, and batch jobs with automatic scaling — designs routing to keep requests sticky to the right execution environment. A first-hand engineering writeup relevant to anyone building serverless GPU inference infrastructure.
Related event: Modal Ships Sticky Sessions for Serverless Routing, Revives Curvytron(2 posts)→
More from Infra
- xAI said to lean on ACC/AEC and copper as Chinese pluggables stay off-limits — jwt0625 · 2026-10-21
- Packed a16z SF infra event features fal, ElevenLabs, BFL and Krea engineers — venturetwins · 2026-10-21
- Baseten ships DeepSeek v4.1 Flash, Reflection Beam 1.0, Nemotron Super 3.5 VL day-zero with claimed top benchmarks — baseten · 2026-10-21
- EdgeKernelBench: LLM-Generated Kernels Hit 64.5% Correct but Only 25.5% Fast on Edge Backends — encodelab · 2026-10-21
- Meta Open-Sources Dist-MoE, Speeding MoE Training Up to 1.40x on GB300 NVL72 — ezyang · 2026-10-21
- PyTorch accelerator integration group details progress and priorities at PyTorchCon — PyTorch · 2026-10-21