Training linear-attention Laplace 3B on AWS Trainium2: 120K tokens/sec, fixed 29.5MB state
AWS ML Blog · rss · 2026-10-21
AWS ML Blog details how i14 trained and served Laplace 3B, a 24-layer model replacing softmax attention with Gated Linear Attention (GLA), on AWS Trainium2 using native PyTorch via TorchNeuron Native.
Architecture
- GLA compresses history into a fixed 29.5 MB recurrent state per sequence, independent of context length; training attention drops from O(T²) to O(T)
- 16 heads with geometrically scheduled decay rates (half-lives from 1 to 2,048 tokens), trained with the Muon optimizer
Three Trainium2 adaptations
- Rewrote the GPU-specific Triton GLA kernel with standard PyTorch operators, using 16 chunks of 128 tokens to match the NeuronCore's 128 on-chip buffer partitions
- Redesigned Muon communication: two alltoall collectives in MuonFSDP2 cut per-rank traffic 24x and raised throughput 1.63x
- Statically unrolled the 16-step GLA recurrence to fit the static compiler's graph and on-chip memory limits
Results: a single trn2.48xlarge instance (16 Trainium2 chips) trains at 120K tokens/sec and serves 11,544 output tokens/sec across 1,280 concurrent sequences.
More from Infra
- Shotwell: Starlink now contracted for ~12,000 planes, a quarter of available aircraft — XFreeze · 2026-10-21
- Samsung Electro-Mechanics signs $260M substrate deal with Micron as AI shortage spreads down the stack — demian_ai · 2026-10-21
- David Ondrej interviews Ahmad Osman on running AI fully locally, DeepSeek 4.1 Flash and DGX Spark vs Mac Studio — David Ondrej · 2026-10-21
- From 9 to 48 tokens/s: STRATA + CachyOS supercharges local Qwen Flash on an 8GB GPU — Ne00n · 2026-10-21
- New Jersey launches $125M+ supercomputer initiative to double university compute — Prof_DavidBader · 2026-10-21
- ASI raises $325M Series C to build AI that controls critical physical infrastructure — darian314 · 2026-10-21