One sparse paged GQA decode kernel written six ways: raw CUDA beats PyTorch eager by 14x on RTX 4060
Daemontatox · reddit · 2026-10-22
The author implemented the same sparse paged GQA decode attention kernel (FP8 KV cache, Top-K 512, page size 16, 256 requests) six ways on an RTX 4060 Laptop (SM89/Ada)—Raw CUDA, Triton, two CuTe DSL variants, Mojo (MAX), and eager PyTorch—and open-sourced all code, charts, and Nsight Compute reports.
Results:
- Raw CUDA C++: 2.164 ms (0.992 TFLOP/s), fastest
- Triton: 2.442 ms, only 13% slower than CUDA; lowest occupancy but highest SM throughput (86.31%)
- Optimized CuTe DSL: 4.441 ms vs baseline CuTe at 15.45 ms—the grouped mapping makes CuTe 3.5x faster (2,048 CTAs instead of 8,192, no shared-memory barriers for QK)
- Eager PyTorch: 31.02 ms; every fused kernel dramatically outperforms it
- Mojo (MAX) not ranked due to host-timer enqueue cost
Observation: occupancy alone doesn't predict latency.
Open problem: grouped kernels don't share K/V loads across the 4 query heads in a CTA—the logical K/V request is 1 GiB vs 256 MiB for eager gather—and the author found no low-risk fix, inviting suggestions. Single-run, laptop-GPU, synthetic-data limits are acknowledged.
More from Infra
- Micron exec insights suggest HBM supply stays tight through 2029 — BenBajarin · 2026-10-23
- C multithreading experiment: padding counters to 64 bytes yields 3-4x speedup via false sharing fix — cneuralnetwork · 2026-10-23
- Two £300 Snapdragon mini PCs, no GPU: 46 tok/s local Qwen3.8 Flash Next with custom ToRc engine — WorldlyAd7946 · 2026-10-22
- ai& runs MiniMax H3 across NVIDIA, AMD and Tenstorrent, undercutting official pricing — DavidBennett__ · 2026-10-22
- With PS5 and Switch 2 jailbreaks taking off, could consoles run local LLMs? — cute_spider · 2026-10-22
- Jensen Huang defends neocloud financing: high borrowing costs are a side effect of huge AI demand, not a business flaw — rohanpaul_ai · 2026-10-22