Rust inference engine beats SGLang 13.4x on 250K-context agent TTFT, open-sourcing soon
ahstanin · reddit · 2026-11-04
The steffi.ai team built llminfra, a Rust-based inference engine for on-device multi-agent systems, and benchmarked it against SGLang on identical RTX PRO 6000 (300W) hardware running Qwen 3.8 27B:
- Turn TTFT at 250K context: 108ms vs 1,440ms (13.4x) — critical for agent loops where context stays loaded and only tool-call observations append.
- Cold start to ready: 6.1s vs 211.2s (34.8x); first token after start 28ms vs 528ms.
- Throughput: 209 vs 95 tok/s with 32 agents at 32K context (2.2x); 86 vs 28 tok/s with 2 agents at 250K (3.1x).
- Peak VRAM: 52.9 GiB vs 86.3 GiB on 8x32K.
The repo is being cleaned up for open-source release; the author self-deprecatingly calls these "trust me bro" benchmarks and invites independent replication.
More from coding & agent
- Free no-sign-in MCP server compiles LaTeX (pdflatex/xelatex) and explains errors in plain words — PurplelinkPL · 2026-11-04
- Multi-agent review pipeline: Claude coordinates, Grok reviews, Codex fixes — PawelHuryn · 2026-11-04
- WoW x League of Legends Arena mashup: roguelike 2v2v2v2 PvP with per-round drafts — TAbrodi · 2026-11-04
- Kimi K3.1 Pro weights drop tomorrow: 2.8T params, 1M-token context — togethercompute · 2026-11-04
- 86% of orgs run AI agents, but under 18% can automatically stop a misbehaving one — amasad · 2026-11-04
- CMU Study: Harness Engineering Boosts Scientific Agents Without Retraining — niloofar_mire · 2026-11-04