CUDA gotcha: default 8 work queues can serialize GPU tasks via false dependencies
AlpinDale · x · 2026-10-26
AlpinDale shared a screenshot from NVIDIA's CUDA docs highlighting an easily missed detail: the CUDADEVICEMAXCONNECTIONS env var controls concurrent compute/copy work queues (1-32, default 8). Independent tasks from different CUDA streams mapping to the same queue create false dependencies that serialize GPU work.
Key points:
- NVIDIA recommends setting work queues >= active CUDA streams per context (e.g. CUDADEVICEMAXCONNECTIONS=16)
- It also modifies copy connections unless CUDADEVICEMAXCOPYCONNECTIONS is set explicitly
- The copy-specific variable only applies to compute capability 8.0+ and overrides the general one
More from Infra
- Running full DeepSeek V4.1 Flash weights on 4x DGX Spark (~$28k): 1M context at 66 tok/s — ricklamers · 2026-10-26
- Meta files $35B stock and mandatory convertible offering, plus $60B ATM program — RihardJarc · 2026-10-26
- "The grid and your home are the bottleneck": compute buyer goes bullish on ARM chips — TheZachMueller · 2026-10-26
- On-device VLM goes production: GenieX runs Liquid AI's LFM-VL live at PyTorch Conference — PyTorch · 2026-10-26
- a16z: OpenRouter token demand up ~27x in a year, open-weight models now serve 72% — TheZachMueller · 2026-10-26
- Giraffe: Open-Source Baseline Testing for OpenAI-Compatible LLM Endpoints — ImmediateBig8479 · 2026-10-26