Engine-Agnostic Rust LLM Gateway SMG Graduates from LightSeek Foundation
vllm_project · x · 2026-08-01
SMG (Shepherd Model Gateway), an LLM inference gateway incubated by the LightSeek Foundation, has officially graduated to an independent open-source community and been accepted into the @PyTorch Ecosystem.
Key features:
- Engine-Agnostic: Built in Rust, it offers full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, SGLang, OpenAI, and more.
- Advanced Routing: Features an industry-first gRPC pipeline, KV cache-aware routing, and tokenization caching.
- Full Integration: Supports chat history, Responses API, WASM plugins, MCP, and multi-tenant auth.
The vLLM team collaborated to standardize frontend components, like multimodal processors, to be shared across open-source projects.
More from Infra
- PyTorch 2.13 Brings FlexAttention to Apple Silicon, Cuts Peak Memory by 4× — PyTorch · 2026-08-01
- Meta Engineer Shares MLSys Keynote: Using AI to Liberate Systems Researchers — salykova_ · 2026-08-01
- Amazon Stock Soars Record 15%, Prediction Markets Bet 2026 Capex Over $200B — Polymarket · 2026-08-01
- StudyFetch Cuts AI Inference Costs ~10x with NVIDIA Riva — nvidia · 2026-08-01
- Building a 4x3090 Local AI Workstation: Still Worth It in 2026? — davyjones10Y · 2026-08-01
- Intel Opens Up CDNA5 Instruction Set Architecture — xiaosun86 · 2026-08-01