Confidential vLLM inference in practice: full-speed GPUs, encrypted NVLink at 645+ GB/s
kbesplv · reddit · 2026-10-26
The warmctl team demonstrates running LLM inference in confidential computing mode on RTX PRO 6000 and 8×B300 GPUs, aiming at "private AI on a neocloud without trusting the neocloud" amid a wave of October 2026 cloud breaches.
Measured results:
- GPU runs at full speed
- CPU↔GPU bridge at 18 GB/s
- Encrypted NVLink at 645+ GB/s
- Key gotcha: vLLM's default config silently produces garbage in confidential mode; a single environment variable fixes it
The post argues for a zero-trust posture toward neoclouds using confidential computing. Full write-up on their blog.
More from Infra
- Critical pre-boot RCE in EDK II PXE driver: one crafted packet hijacks servers before Secure Boot — cyb3rops · 2026-10-28
- Screenless M1 Airs at $30 per 1,000 multicore points spark cheap Mac cluster idea — vincent_koc · 2026-10-28
- RTX 3080 Tensor Core unlock claims 80-87% TFLOPS boost, RTX 50 series blocked — Revolutionalredstone · 2026-10-28
- CXMT to start small-scale HBM3E production within two months, yield at 20% — zephyr_z9 · 2026-10-28
- Video Transcription Costs Under 1 Yuan per Hour, CPU Free but Error-Prone — huangyun_122 · 2026-10-28
- A forgotten supercomputer design: bare dies, gold post interconnects, 3D stacking, immersion cooling — jwt0625 · 2026-10-28