openjev-sglang: Qwen3.6-35B-A3B serves Jev-compatible API, 64 tasks in under 1s
andersonbcdefg · x · 2026-09-18
Developer ekzhang open-sourced openjev-sglang, an HTTP API implementation compatible with TypeSafe/Jev, backed by Qwen3.6-35B-A3B.
- Performance: Uses SGLang's radix cache to preserve prefill reuse, achieving extremely fast parallel generation — 64 tasks complete in under 1 second.
- Architecture: One B200 per container running SGLang 0.5.19's Rust frontend with radix caching and breakable prefill CUDA graphs; a separate Python API process (FastAPI + uvloop + Rust-backed HF tokenizer) connects to local SGLang via pooled async HTTP.
- Deployment: One-command deploy on Modal via uv run modal deploy modalapp.py, yielding a public autoscaled endpoint; all CUDA deps stay in the SGLang container.
Related event: Developer ships Jev-compatible self-hosted API on Qwen3.6(2 posts)→
More from coding & agent
- Tencent unveils Hyra 2.0, a research agent that builds reusable scientific tools — TencentHunyuan · 2026-11-02
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- "Coding isn't the bottleneck anymore — usage windows are," says founder Kyle Gawley — kylegawley · 2026-11-02
- Demo passed ≠ production-ready: 5 questions to answer before letting agents run your business — _kellhus · 2026-11-02
- An AI agent passed 100 test chats—still not ready to run the business — djgoel123 · 2026-11-02
- Agent given total creative freedom produces a 1:44 code-drawn animated short film — ivegotnoidea1 · 2026-11-02