YC Paper Club on inference: every training FLOP costs ~6x per year in serving, a $220B stack
ycombinator · x · 2026-10-25
Y Combinator's Paper Club digs into the LLM inference stack with builders from Together, Ollama, Luminal, and SGLang. Key framing: for every FLOP spent training, roughly six go to serving each year — and getting inference right has produced over $220B in market cap over the past decade.
The session walks a token from API request to GPU and back, covering routing, caching, scheduling, kernels, and memory. Highlights: Dan Fu on the lifecycle of an inference request; Ollama's Jeff Morgan on why the same open model performs differently depending on who serves it; Luminal's Joe Fioti on compiling models with search instead of heuristics; SGLang's Liangsheng Yin on memory management for the agent era. Full timestamps included.
Related event: YC Paper Club: Inference Costs Six Times Training Per Year(3 posts)→
More from Infra
- Samsung reportedly using TSMC 3nm for Nvidia's next-gen NVHBM custom chip base dies — Beth_Kindig · 2026-10-25
- No AI bubble yet: revenue per GPU minus rental is the real equation — PTrubey · 2026-10-25
- AI labs are flipping GPU contracts: H100/B200 rental prices up 2-3x since early lock-ins — Yuchenj_UW · 2026-10-25
- Swarms open-sources RouteHub, an LLM gateway that imports 190x faster than litellm — KyeGomezB · 2026-10-25
- Radeon 9700 Prices Double in a Month, Hurting Personal Inference Server Builds — UltramanQuar · 2026-10-25
- New blog demystifies GPU communications, from page tables to NVLink, via SGLang internals — BanghuaZ · 2026-10-25