Deep-dive worklog: optimizing CUDA GEMM from naive kernel to shared-memory tiling
abhijithneil · x · 2026-09-22
Independent researcher Anshuman Mishra published part 4 of his 'How to make your models fast' series: a detailed worklog on making CUDA GEMM fast, with open-sourced code.
Key flow:
- Starts from a naive one-thread-per-output kernel, quantifying FLOPs (2MNK-MN) and memory requests to show the bottleneck is memory, not compute
- Introduces online softmax, implements the kernel, and measures memory/FLOPs gains
- Applies coalesced memory access patterns and quantifies improvements in FLOPs, comms and memory
- Reuses tiles from shared memory and computes multiple outputs per thread
Each step follows a quantify-implement-verify loop, making it a practical hands-on guide into inference engineering and CUDA kernel optimization.
More from Infra
- OpenCV launches Enterprise support from $150k/year, library stays open source — mrpollo · 2026-09-22
- Google Cloud previews cross-cloud caching for borderless Lakehouse to cut transfer bills — rseroter · 2026-09-22
- VRAM vs Blackwell Speed: 3090 Ti or 5080 for Local Video Generation? — Apollorax · 2026-09-22
- Liquid AI's LFM2.5 tops mobile benchmarks: 2.32GB memory, 8s latency on iPhone 17 Pro — maximelabonne · 2026-09-22
- AMD Engineers Publish GEMM Optimization Tutorial Blog Inspired by the GEMM Ladder — simran_s_arora · 2026-09-22
- Why Meta hasn't shipped Muse in WhatsApp: not enough hardware for 2B users, says user — zephyr_z9 · 2026-09-22