vLLM offloads video decoding to NVDEC, 2x+ throughput on 8×H100
vllm_project · x · 2026-09-22
vLLM integrates PyNvVideoCodec to fix the CPU bottleneck in large-scale video captioning
- Previously vLLM could only decode video via CPU-side OpenCV+FFMPEG; with short outputs (100-200 tokens), decoding dominated and CPU cores maxed out with just 2-4 GPUs
- By offloading decoding to NVIDIA's hardware NVDEC units via PyNvVideoCodec, throughput rises 2x+ on 8×H100 and the CPU bottleneck is gone, scaling well up to 8 GPUs
- Target use cases: AV training video description and searchable metadata; ships with CUDA vLLM releases
More from Infra
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Eric Schmidt's foundation funds French NGO fighting against data centers — StephanSturges · 2026-11-12
- Epoch: Open-Weight GLM 5.3 Hacks Six Firefox Versions for ~$1,000 Each; Claude Fable 5.2 Sets ECI Record — Epoch AI · 2026-11-12
- Polymarket prices 28% odds of a state data center moratorium; Torus raises $500M — Polymarket · 2026-11-12
- AWS-wrapper cloud startup shuts down just one month after seed round — thedealdirector · 2026-11-12
- Utah startup Torus raises $500M for shipping container-sized AI data centers deployable in 4 months — Polymarket · 2026-11-12