TrnRT: Open-Source Inference Engine Serves GLM-5.3 on AWS Trainium at 3.57M TPM

sanchitmonga22 · hn · 2026-11-18

RunAnywhere (YC W26) open-sourced TrnRT, an LLM inference engine for AWS Trainium, built in three weeks. It serves two GLM-5.3 flash (FP8) instances on one trn2.48xlarge with a cache-aware router, hitting 3.57M TPM on SemiAnalysis's AgentX benchmark, 28.9 RPM with 32 agents, and 1.24s P50 TTFT at 35.9 tps decode. Decode runs as one NKI kernel per layer half, fusing both blocks for 11 sparse-attention layers. Motivation: Trainium memory costs less than half per TB vs 8x H200, while AWS's own stack (vLLM Neuron, NxDI, SGLang Neuron) doesn't support the model.

Original post →

More from coding & agent

coding & agent channel →