Writing LLM Inference CUDA Kernels From Scratch
Abhishekcur · x · 2026-07-13
This reply shares a hands-on guide to implementing and optimizing LLM inference kernels from scratch. The author rewrote several core operators using CUDA to gain a more intuitive understanding of the decode phase's compute patterns.
What was done
- Covered operators include: GEMV, RMSNorm, SiLU, softmax, attention.
- Each operator was optimized step-by-step from a naive implementation to a more efficient version.
- All implementations passed 161 correctness checks and were aligned with a fp64 CPU reference.
Key Observations
- On an RTX 3060, GEMV achieved about 92% of peak DRAM bandwidth.
- However, it only utilized around 1% of the compute power, proving that the decode phase is more of a memory game than a FLOPS game.
- The author's favorite part was related to flash-decoding, highlighting how crucial such optimizations are for actual inference.
Overall, this is a highly technical, engineering-focused share on learning and reproducing inference kernels.
Related event: Developer Builds LLM Inference CUDA Kernels from Scratch(3 posts)→
More from Infra
- Devin adds e2b sandboxes for remote agent execution — badphilosopher · 2026-07-22
- Arbitrum fee simulation shows higher gas capacity but lower L2 revenue under ArbOS61 — tomwanhh · 2026-07-22
- NVIDIA pushes OpenUSD as the common layer for simulation and physical AI — MonaJalal_ · 2026-07-22
- SkyPilot exits stealth with $20M to unify fragmented GPU compute across five clouds — skypilot_org · 2026-07-22
- Production AI budgets include retries, routing, caching and observability—not just token prices — arx-go · 2026-07-22
- NVIDIA briefs analysts on Vera CPU and doubles down on monolithic agentic design — BenBajarin · 2026-07-22