Torch Compile underrated: CUDA Graphs + PDL hits 98% of megakernel speed on Llama 1B

simran_s_arora · x · 2026-11-17

Another thread from Dylan Lim: Torch Compile's generality and performance are underappreciated — default Inductor kernels, CUDA Graphs, and fine-grained PDL placement reach 98% of their previous Llama 1B megakernel performance.

Chart data on Llama 1B / H100 / batch size 1: PyTorch eager 26 TPS → torch.compile 669 → +CUDA graphs 864 → +PDL 953 → +fine-grained PDL 975 TPS (megakernel: 990 TPS). But Torch Compile has fundamental shortcomings: compiler heuristics can't reliably pick the best implementation across workloads, isolated kernel measurements don't reflect e2e speed, and strict rules leave performance on the table.

Original post →

More from coding & agent

coding & agent channel →