First PyTorch Madrid meetup demos llm-d: K8s-native distributed LLM inference on vLLM
SergioPaniego · x · 2026-10-15
The first PyTorch Madrid meetup featured a deep dive into the open AI inference stack: vLLM and llm-d, distributed inference, and smarter GPU orchestration on Kubernetes. Slides show llm-d is an open-source, Kubernetes-native project that adds smart orchestration on top of vLLM — auto-scaling prefill (compute-optimized) and decode (memory-optimized) GPU pools separately, with an inference-aware gateway routing requests; HuggingFace models run natively without conversion (github.com/llm-d).
More from Infra
- Thinky's Dimuon FAQ: no RL gains over AdamW, tune learning rate only — eliebakouch · 2026-10-15
- vLLM-Metal merges sleep mode: frees KV cache memory on idle Macs without restart — abhijithneil · 2026-10-15
- Neocloud inference debate: 95% GRR and 400% NDR beat 'reverse auction with one-line API change' — Ronangmi · 2026-10-15
- Gumroad cut $100k/year from AWS by moving one Rails webhook endpoint to Go on 3 servers — DanielLockyer · 2026-10-15
- Anthropic's margins imply 70-75% of its compute goes to R&D, analysis shows — Jsevillamol · 2026-10-15
- Tsinghua's Latent Compaction Compresses KV Cache 32× While Keeping 96.7% Retrieval Accuracy — Yingfa Chen · 2026-10-15