Inference practitioner on GPU co-ops: saturation math, weight caching, and quantization as dark art
sloppenheimer · x · 2026-11-23
Replying to a question about renting GPUs for open-model inference, sloppenheimer shares hard-won practice: his team serves open models under ZDR, optimizes for Blackwell, and even works on TPM-signing weights. Renting makes sense if you can saturate GPUs; otherwise offset costs with fast startup — cached weights, precompiled CUDA graphs, minimal load times. Good quantization and specdec drafters remain art more than science, and co-ops face challenges well beyond standing up sglang/vLLM: KV cache dynamics, startup/recovery/autoscaling, speculative decoding.
Related event: Practitioner shares lessons on GPU inference co-ops and privacy(2 posts)→
More from coding & agent
- Popfold: an open-source three.js engine built with parallel Claude Code sessions — cherichita · 2026-11-23
- Weaviate Podcast #150 dives deep into rerankers and agentic search — CShorten30 · 2026-11-23
- Job-hunting agent fixes reviewed: idempotency keys help, but AUC stays 0.645 — nivokvo · 2026-11-23
- Siemens and AGIBOT generate robot grippers from text via agentic AI workflow — emmanuelvivier · 2026-11-23
- DeepSeek on 4 DGX Sparks codes great with Opus 5.5 planning, GLM-5.3 not ready — AIFlow_ML · 2026-11-23
- Made a Video for $0.09: Full Workflow with Claude + VidLab API — eptwts · 2026-11-23