CoreWeave inference chief: 80-90% of agentic input is repeat, so cache shapes the whole stack
AI Engineer · youtube · 2026-09-20
CoreWeave's inference lead Sitanshu Gupta explains the platform's architecture from MVP to trillion-parameter workloads. Key insight: 80-90% of an agentic request's input is identical to the previous one, and prefill is the most expensive part of inference — hence cached tokens are priced far below fresh ones. The platform serves serverless (per-token, plus a provisioned throughput tier) and dedicated (per-GPU-hour, customer-owned deployment) without forking. Four workload shapes fit like Tetris across time: agentic/chat (latency-tight), streaming voice/video, and 12-hour batch jobs drained overnight on dedicated capacity. The router prioritizes KV cache locality first, least-loaded second; KV cache is offloaded to high-bandwidth storage between turns. Biggest recent levers: 4-bit quantization and speculators trained asynchronously on customer data.
More from Infra
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04
- Intel may discontinue Arc B580 to prioritize higher-margin Arc Pro cards as B70 prices rise — chemist_slime · 2026-11-04