openjev-sglang: Qwen3.6-35B-A3B serves Jev-compatible API, 64 tasks in under 1s
andersonbcdefg · x · 2026-09-18
Developer ekzhang open-sourced openjev-sglang, an HTTP API implementation compatible with TypeSafe/Jev, backed by Qwen3.6-35B-A3B.
- Performance: Uses SGLang's radix cache to preserve prefill reuse, achieving extremely fast parallel generation — 64 tasks complete in under 1 second.
- Architecture: One B200 per container running SGLang 0.5.19's Rust frontend with radix caching and breakable prefill CUDA graphs; a separate Python API process (FastAPI + uvloop + Rust-backed HF tokenizer) connects to local SGLang via pooled async HTTP.
- Deployment: One-command deploy on Modal via uv run modal deploy modalapp.py, yielding a public autoscaled endpoint; all CUDA deps stay in the SGLang container.
More from coding & agent
- GitHub rewrote the Copilot runtime in 800,000 lines of Rust, mostly by AI agents — DanWahlin · 2026-09-18
- Bolt Forge adds Kimi K3, DeepSeek V4 Pro and GLM 5.3 — Flash variant leads with 54% usage — SucceededMind · 2026-09-18
- Paying a FasTrak toll invoice automatically with a Grok bot and Link — jeff_weinstein · 2026-09-18
- Honeycomb CTO on AI slop: half the company hates it, half hates the holdouts — mipsytipsy · 2026-09-18
- Credential-free MCP server template lets agents call tools without touching API keys — uzi24- · 2026-09-18
- openwiki v0.5.2 adds IBM bob as its second coding agent integration, now 6 total — Hacubu · 2026-09-18