FLAT brings open-source linear attention training kernels to TPUs, up to 3x faster
SonglinYang4 · x · 2026-10-14
Researchers from UC Berkeley, Stanford, Princeton and Together AI (including Percy Liang, Ion Stoica, and Tri Dao) released FLAT: Flash Linear Attention on TPUs, among the first open-source kernels for training linear attention layers on TPUs.
- Production LLMs now widely use linear attention variants: Kimi K3 uses Kimi Delta Attention, and Qwen 3.5 interleaves Gated DeltaNet layers with full attention. Linear attention replaces the growing KV-cache with a fixed-size recurrent state, giving constant time/memory per token and O(T) training cost.
- GPUs already have fast kernels (FLA, Mamba CUDA), but TPU work so far targeted inference only; no open-source training kernels existed.
- FLAT provides Pallas implementations of Linear Attention, Mamba-2, and Gated Linear Attention, benchmarked on a TPU v6e (Trillium), plus a fast kernel for the depthwise causal short convolution preceding the recurrence.
- Counting forward and backward passes, FLAT on one TPU v6e chip is up to 3x faster. Blog and code are open source; delta-rule kernels (Gated DeltaNet, KDA) are next.
More from Infra
- On-device web search for a 1.7B phone-local LLM, no search API or backend — K4anan · 2026-10-14
- Qwen 3.8 27B served at ~300 tokens/second on Nebius, claims fastest deployment yet — Arindam_1729 · 2026-10-14
- llama.cpp adds --spec-draft-n-ubatch option to optimize VRAM usage — jacek2023 · 2026-10-14
- Two small LLMs on a Galaxy S21 complete an Amazon shopping task locally in 51 minutes — Mean-Standard7390 · 2026-10-14
- RouterPlus pools demand so small teams get bulk-rate LLM inference pricing — rohanpaul_ai · 2026-10-14
- Adding €300 of used RAM for 80GB to run MoE models: is it worth it? — quadra-lab · 2026-10-14