CoreWeave inference chief: 80-90% of agentic input is repeat, so cache shapes the whole stack
AI Engineer · youtube · 2026-09-20
CoreWeave's inference lead Sitanshu Gupta explains the platform's architecture from MVP to trillion-parameter workloads. Key insight: 80-90% of an agentic request's input is identical to the previous one, and prefill is the most expensive part of inference — hence cached tokens are priced far below fresh ones. The platform serves serverless (per-token, plus a provisioned throughput tier) and dedicated (per-GPU-hour, customer-owned deployment) without forking. Four workload shapes fit like Tetris across time: agentic/chat (latency-tight), streaming voice/video, and 12-hour batch jobs drained overnight on dedicated capacity. The router prioritizes KV cache locality first, least-loaded second; KV cache is offloaded to high-bandwidth storage between turns. Biggest recent levers: 4-bit quantization and speculators trained asynchronously on customer data.
More from Infra
- Nscale revenue up 1,252% yet loses over $1 billion in six months — TansuYegen · 2026-09-20
- HilbertRaum: open-source app packs local models, docs and chats onto a USB drive — Vladowski · 2026-09-20
- 500K Embedding Tokens/Sec on One GPU: Superlinked's Small-Model Serving Stack — AI Engineer · 2026-09-20
- Entity resolution with Jev cut pipeline costs 99.56% and boosted throughput 7.35x — hrishioa · 2026-09-20
- Home 8x RTX 6000 Pro rig now costs as much as a condo, warns builder — TheZachMueller · 2026-09-20
- SGLang x Datawhale Add New Chapters to Open-Source Inference Engine Course — ying11231 · 2026-09-20