Inference practitioner on GPU co-ops: saturation math, weight caching, and quantization as dark art

sloppenheimer · x · 2026-11-23

Replying to a question about renting GPUs for open-model inference, sloppenheimer shares hard-won practice: his team serves open models under ZDR, optimizes for Blackwell, and even works on TPM-signing weights. Renting makes sense if you can saturate GPUs; otherwise offset costs with fast startup — cached weights, precompiled CUDA graphs, minimal load times. Good quantization and specdec drafters remain art more than science, and co-ops face challenges well beyond standing up sglang/vLLM: KV cache dynamics, startup/recovery/autoscaling, speculative decoding.

Related event: Practitioner shares lessons on GPU inference co-ops and privacy(2 posts)→

Original post →

More from coding & agent

coding & agent channel →