Implementing LLM Inference CUDA Kernels From Scratch
Abhishekcur · x · 2026-07-13
The author implemented and optimized LLM inference-related kernels from scratch using CUDA, covering GEMV, RMSNorm, SiLU, softmax, and attention. All 161 correctness checks passed and aligned with the fp64 CPU reference implementation.
Testing on an RTX 3060 showed that GEMV reached about 92% of peak DRAM bandwidth but used only 1% of compute power, indicating that the decode phase is primarily a memory bandwidth issue rather than a FLOPS issue.
The author's favorite part was implementing flash-decoding from scratch: online softmax + split-KV achieved about 80% of peak performance at a 16K context, while the naive implementation couldn't even launch due to shared memory limits. They also noted a 210x speedup from naive softmax to the optimized version on a logits-scale matrix. Next steps include implementing bf16, utilizing tensor cores, and conducting more comprehensive profiling.
Related event: Developer Builds LLM Inference CUDA Kernels from Scratch(3 posts)→
More from coding & agent
- FactoryAI gave back its first millions, then shipped Droid CLI two years later — matanSF · 2026-07-22
- Devin Outposts aims to run AI agents on any machine, from Mac minis to Kubernetes clusters — blaizedsouza · 2026-07-22
- Hermes Agent Refactoring Proposal: Decoupling via Event Bus and Monorepo Slicing — Promptmethus · 2026-07-22
- ty now reads Pydantic config keywords and field metadata — charliermarsh · 2026-07-22
- Pensar Launches AI Security Agent to Autonomously Discover and Patch 0-Days — andriy_mulyar · 2026-07-22
- ty adds first-class Pydantic support, including strict and lax field handling — charliermarsh · 2026-07-22