AI Compilers Are Redefining Execution Boundaries
青稞AI · wechat · 2026-07-18
This article systematically reviews the evolution of AI compilers and inference/training infrastructure. The core argument is: the traditional operator-bound compilation paradigm is no longer sufficient. Current work is pushing execution units down to the tile/task level, deepening semantic fusion, and incorporating communication into the IR. Key sections include:
1) Compilation boundaries shifting from operators to tiles/tasks
The author argues that workloads like LLMs, long context, KV cache, MoE, dynamic batching, and distributed training make the "one operator per boundary" abstraction too coarse.
Representative directions mentioned include:
- TensorIR: Elevating tensor computation primitives to first-class objects.
- Welder: Explicitly modeling tile-level data movement and reuse from a memory access perspective.
- Ladder: Incorporating low precision and custom datatypes into the optimization space.
- TileLang / ThunderKittens: Making it easier for developers to write high-performance GPU kernels.
- Cypress: Describing GPU computation via task-based tensor computation, leaving data movement and synchronization to the compiler.
2) Moving from plain fusion to "semantic fusion"
The article notes that traditional operator fusion falls short for patterns with reduction dependencies, like attention. FlashAttention's key breakthrough was using online softmax to bypass the limitation of materializing full scores.
It highlights Neptune:
- First performing a "seemingly illegal" naive fusion;
- Then automatically fixing intermediate reductions via algebraic repair;
- For example, using correction terms like exp(oldmax - newmax) in softmax;
- This approach covers both plain attention and decoding attention, approximating expert techniques like FlashAttention / FlashDecoding.
Also mentioned:
- FlexAttention: Allowing users to express variants like masks, score modifications, and block sparsity with minimal PyTorch code.
- FlashInfer: Designed for LLM serving, combining KV cache, block-sparse formats, JIT attention templates, and load-balancing scheduling.
- Nautilus: Attempting to automatically discover optimization sequences similar to FlashAttention-3 from high-level algebraic specifications.
3) Superoptimizers, megakernels, and dynamic execution
The article attributes the recent resurgence of superoptimizers to the sheer volume of model variants, shapes, dtypes, and attention masks, making it impossible for experts to handwrite all optimal kernels.
Representative works include:
- Mirage: Unifying kernel / threadblock / thread hierarchies using µGraphs, and searching over algebraic transformations, scheduling changes, and custom kernel generation.
- megakernel / persistent kernel: Placing multiple operators or tasks into a single long-running kernel to reduce launch overhead.
- EventTensor: Explicitly encoding dependencies between tasks, enabling dynamic megakernels to handle shape-dependent and data-dependent dynamics.
- Infera: Slicing large operators into tiles/micro-kernels at compile time, combined with runtime state for dynamic scheduling.
The author emphasizes that truly effective optimization isn't just about "searching more," but organizing the search space correctly.
4) Distributed compilers start incorporating communication into the IR
The bottleneck on the training side is often communication, not pure compute. The article summarizes approaches to making communication explicit:
- Concerto: Tracing PyTorch into an FX graph, then constructing an IR containing both compute and communication operators, and scheduling overlaps using resource-constrained project scheduling.
- Triton-distributed: Directly integrating communication primitives into a Triton-style high-level programming model.
- TileLink: Expressing computation-communication overlap in a tile-centric manner.
5) Conclusion
The author's final assessment is:
- AI compilers are shifting from "operator optimization" to "semantic and system co-optimization";
- Attention, megakernels, and communication overlap are all being re-abstracted by compilers;
- The truly impactful directions for the future likely won't be a single point kernel, but rather abstracting expert experience into composable compiler primitives.
More from Infra
- Nebius says SlimSpec speeds speculative decoding 8–9% without shrinking the vocabulary — Arindam_1729 · 2026-07-21
- NVIDIA brings its Cosmos 3 Edge world model to Jetson for on-device robot control — liu_mingyu · 2026-07-21
- A silicon photonic reservoir chip compensates fiber distortion in real time at 28 Gbps — bravo_abad · 2026-07-21
- Chamath says open-sourcing Grok would push AI margins from models to infra and apps — Dan_Jeffries1 · 2026-07-21
- EU AI competitiveness is under pressure as firms double down on chips, ethics, and talent — nordicinst · 2026-07-21
- AI bottlenecks are shifting to memory, optics, yield control and power — thedealdirector · 2026-07-21