Building LLM Inference CUDA Kernels From Scratch
Abhishekcur · x · 2026-07-13
The author shares their experience building LLM inference kernels from scratch using CUDA to truly understand the compute characteristics of the decode phase.
What was done
- Implemented multiple operators: GEMV, RMSNorm, SiLU, softmax, attention
- Each operator went through a process from a naive implementation to an optimized version
- Conducted 161 correctness checks, all passed against an fp64 CPU reference
What was measured
- On an RTX 3060, GEMV achieves around 92% of peak memory bandwidth
- But it only consumes about 1% of the compute resources
- Conclusion: decode is more of a memory bandwidth issue than a FLOPS issue
The author also mentions that their favorite part is the content related to flash-decoding.
Related event: Developer Builds LLM Inference CUDA Kernels from Scratch(3 posts)→
More from coding & agent
- A roundup of AI agents and MCP resources, including how to evaluate agents — _jaydeepkarale · 2026-07-21
- A full course shows how to build and deploy an AI agent with OpenAI and LangChain — _jaydeepkarale · 2026-07-21
- A beginner guide to AI agents points readers to a Stanford webinar — _jaydeepkarale · 2026-07-21
- A practical guide on how to evaluate AI agents — _jaydeepkarale · 2026-07-21
- MCP is headed toward easier scale, event-driven extensions, and workable file uploads — EricBuess · 2026-07-21
- Developers debate the missing composition model for AI agents — threepointone · 2026-07-21