Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline
AndrewLBeam · x · 2026-10-23
The Lila Sciences team implemented 1M-token context training for GLM-5.3-Flash on SkyRL — 64x beyond the out-of-the-box Megatron baseline — via four lines of optimization:
- DeepSeek Sparse Attention: running DSA on a sparse kernel, chunking and sharding its quadratic indexer
- Activation recompute & offloading: recomputing activations in backward and offloading saved inputs to CPU
- Context parallelism: splitting sequences across GPUs, with new CP support for KDA and a DSA fix
- Avoiding unnecessary precision upcasts: keeping FP32 copies out of memory and off the network
A full technical blog covers implementations and results — useful reference for any team doing long-context training or RL.
Related event: SkyRL Extends GLM-5.3-Flash Training Context to 1M Tokens(3 posts)→
More from Infra
- Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
- Perplexity's Portable Computer lands on AMD Ryzen AI Max, running local 27B model — AravSrinivas · 2026-10-23
- Liquid AI Launches RIPL: Ultra-Low-Latency Inference Engine With Sub-50ms End-to-End Responses — JosephJacks_ · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang lays out NVIDIA's full Vera Rubin agentic system: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- ONNX to Transformers.js: running models directly in your browser — JosephJacks_ · 2026-10-23