SGLang-Neuron v0.1.0 open-sourced: SGLang serving stack now runs on AWS Trainium
ying11231 · x · 2026-11-13
SGLang and AWS Annapurna Labs released SGLang-Neuron v0.1.0 as open source in public preview, bringing the SGLang serving stack natively to AWS Trainium2/Trainium3 via a PyTorch-native plugin (same server, same APIs through TorchNeuron, also for RL rollouts). Key features:
- Fully async runtime: Trainium chips run decode steps back to back
- Megakernels: every decoder layer of a step in a single NKI kernel call
- PD disaggregation over EFA: Kimi-K3 across three Trainium3 nodes, GLM-5.2 at 1M context
- HiCache: evicted prefixes stay in host memory and reload on later hits
- SGLang Diffusion on Trainium2: FLUX.1-dev, FLUX.3-Action, MiniMax-H3, Wan2.2 A14B, LingBot-World-V2
A notable step for the non-NVIDIA accelerator software ecosystem.
More from Infra
- SGLang Summit agenda: LinkedIn serves ads model at 10K+ QPS under 80ms — DavidBennett__ · 2026-11-13
- Meme explains CUDA perfectly: optimized for many Alice Huangs, not one — djcows · 2026-11-13
- Perplexity's Mac app shows a local models option, hinting at offline capability — dnlkwk · 2026-11-13
- 2-bit DeepSeek V4.1 Flash runs a coding-agent seat on two 2016 V100s: a 12-hour real-work trial — a4mula · 2026-11-13
- IBM QDC 2026 Unveils Nighthawk r2 Quantum Chip and Fault-Tolerant Dev Tooling — MJBiercuk · 2026-11-13
- Why Jensen Huang Splits From US Frontier Labs on Open Source and Export Controls — morganlai · 2026-11-13