TrnRT: Open-Source Inference Engine Serves GLM-5.3 on AWS Trainium at 3.57M TPM
sanchitmonga22 · hn · 2026-11-18
RunAnywhere (YC W26) open-sourced TrnRT, an LLM inference engine for AWS Trainium, built in three weeks. It serves two GLM-5.3 flash (FP8) instances on one trn2.48xlarge with a cache-aware router, hitting 3.57M TPM on SemiAnalysis's AgentX benchmark, 28.9 RPM with 32 agents, and 1.24s P50 TTFT at 35.9 tps decode. Decode runs as one NKI kernel per layer half, fusing both blocks for 11 sparse-attention layers. Motivation: Trainium memory costs less than half per TB vs 8x H200, while AWS's own stack (vLLM Neuron, NxDI, SGLang Neuron) doesn't support the model.
More from coding & agent
- Chinese devs vent: app store rejections and copycats are just the norm — tinyfool · 2026-11-22
- Compacting Early and Often Boosts Agent Accuracy While Cutting Cost and Runtime — zainhas · 2026-11-22
- Astra 6.2 silently skips modules in audit tasks, then claims it checked everything — MbahSurip · 2026-11-22
- Is Haskell 'unsafe' too? A debate on types vs verification in the LLM era — satnam6502 · 2026-11-22
- Lessons from building a desktop AI agent that sees your screen and drives the mouse — SpeechAggravating453 · 2026-11-22
- MCP server checks if a pay-per-call x402 API is worth paying, from 24,766 real paid calls — biryaniAssasin · 2026-11-22