Before porting code to GPU, measure memory transfer time — data movement is the real bottleneck
Franc0Fernand0 · x · 2026-09-15
Porting code to the GPU doesn't automatically make it faster. The author shares a rule from a GPU programmer: before porting, measure how long memory transfers take by running a dummy program that only moves data to and from the GPU. If transfers cost more than the CPU computation, porting is pointless.
Key points:
- A GPU does three things at once: transfer to CPU, transfer from CPU, and compute. Optimal setups overlap transfers to keep compute busy.
- Chunk your data so transfers "hide" behind computation of the previous block.
- Whether the CPU can do useful work while waiting on GPU results determines if transfer time matters.
The lesson generalizes: measure the cost of moving data before optimizing compute — data movement is often the real bottleneck.
More from Infra
- Own your embeddings: Weaviate + Ollama local vector search config that never leaves your network — philipvollet · 2026-09-15
- Chatbot injects 252k-token prompt every turn instead of RAG, dev asks to be talked out of it — louay_Sallakho · 2026-09-15
- MinIO: open-source S3-compatible object store with zero egress fees — thisguyknowsai · 2026-09-15
- Go-LLM repo details running Flash-Next on dual RTX 3090 via vLLM, with honest benchmarks — Motor_Ad16 · 2026-09-15
- MediaTek's Dimensity 9600 Pro: 2nm chip doubles AI compute, cuts power — jiqizhixin · 2026-09-15
- Tuning SGLang on a single 5090 for Qwen3.8-27B: 100 tok/s but only 82k context — ni1by2thetrue · 2026-09-15