SkyRL and Lila Sciences scale GLM-5.3-Flash training to 1M-token context, 2.1x faster at 256K
AndrewLBeam · x · 2026-10-23
The Anyscale SkyRL team partnered with Lila Sciences' AI research team to push GLM-5.3-Flash training to 1M-token sequences (64x the previous baseline) with 2.1x faster steps at 256K, with all optimizations open-sourced and several fixes upstreamed to Megatron-LM.
Key techniques:
- DeepSeek Sparse Attention (DSA): run on a sparse kernel with the quadratic indexer chunked and sharded — the authors found "the implementation of sparse attention wasn't so sparse."
- Activation recompute and offloading: recompute activations in backward, offload saved inputs to CPU.
- Context parallelism: sequence sharding across GPUs, with new CP support for KDA and a fix for DSA.
- Avoiding precision upcasts: keeping fp32 copies out of memory and off the network.
Caveat: throughput/memory numbers come from dummy fine-tuning on random tokens; real-world uneven MoE routing raises peak memory (covered in the blog). A practical reference for teams training long-context agents.
Related event: SkyRL Extends GLM-5.3-Flash Training Context to 1M Tokens(3 posts)→
More from Infra
- Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
- Scaling GLM-5.3-Flash training to 1M token context, 64x over Megatron baseline — AndrewLBeam · 2026-10-23
- Perplexity's Portable Computer lands on AMD Ryzen AI Max, running local 27B model — AravSrinivas · 2026-10-23
- Liquid AI Launches RIPL: Ultra-Low-Latency Inference Engine With Sub-50ms End-to-End Responses — JosephJacks_ · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang lays out NVIDIA's full Vera Rubin agentic system: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23