SGLang-Neuron v0.1.0 open-sourced: SGLang serving stack lands on AWS Trainium
testingcatalog · x · 2026-11-18
LMSYS's SGLang team and AWS Annapurna Labs released SGLang-Neuron v0.1.0, an open-source PyTorch-native plugin serving LLMs and diffusion models on AWS Trainium.
- One fully asynchronous runtime for all supported LLMs; Megakernels run every decoder layer of a step in a single NKI kernel call (GPT-OSS-120B, GLM-5.2, Kimi-K3).
- Kimi-K3 runs prefill-decode disaggregated across three Trainium3 nodes with KV over NIXL on EFA; GLM-5.2 (MXFP4) serves 1M-token context for agentic coding.
- HiCache keeps evicted prefixes in host DRAM for reload instead of recompute — yielding 70%+ more output for GPT-OSS-120B when chip memory fills.
- SGLang Diffusion runs on Trainium2 (FLUX.1-dev, FLUX.3-Action, Wan2.2 A14B, etc.), plus RL rollouts with Miles.
A notable step for the non-NVIDIA inference ecosystem and open-model serving economics on Trainium.
More from Infra
- Blackstone warns Anthropic will debut as junk-rated after record IPO, clouding $63B chip financing — deliprao · 2026-11-18
- Japan's supercomputer Fugaku hacked, nearly 4,600 users' credentials at risk, now offline — saibharadwaj · 2026-11-18
- Microsoft open-sources bitnet.cpp: 100B-parameter 1-bit LLM on a single CPU, 82% less energy — dr_cintas · 2026-11-18
- Harvard: 400 largest US data centers emit as much greenhouse gas yearly as Denmark — geoffwolfe · 2026-11-18
- TomorrowX pitches: open source closing on frontier models, Nvidia's software moat is gone — PTrubey · 2026-11-18
- Astera Labs CEO: demand outlook unchanged, customers just want product faster — BenBajarin · 2026-11-18