SGLang Adds /v1/score and Multi-Item Scoring, Cuts 16-Candidate p95 to 20.6ms
hsu_byron · x · 2026-09-26
LMSYS published a blog on scaling JEV-like decision models with SGLang. Serving score-based decision models with Generate + top-k logprobs risks dropping the needed label and recomputing shared context per candidate. SGLang's new /v1/score endpoint returns scores for exact requested labels, while Multi-Item Scoring (MIS) computes the shared context once with candidates isolated. MIS latency stays nearly flat from 2 to 16 candidates: on Qwen3-8B, 16-candidate p95 drops from 54.1ms to 20.6ms, and stays under 100ms under load on Qwen3-0.6B vs seconds for alternatives. Contributed by LinkedIn.
More from coding & agent
- Dev says Slack Claude writes over 50% of his PRs, but it's costly without a subscription — kavirkaycee · 2026-09-26
- Grok launches Bot Creator Rewards; one creator reports $500 in two weeks — billyuchenlin · 2026-09-26
- OpenRouter open-sources Jev Router, a cache-aware router that picks the best model per request — markjeffrey · 2026-09-26
- 4B model mines an iron pickaxe in Minecraft in 23 decisions, generating zero tokens — Top-Evidence174 · 2026-09-26
- Open-source Jev router cuts LLM token costs ~9x by routing Claude Code and Codex traffic — airesearch12 · 2026-09-26
- Mica v0.1 4B: Open Decision Model Runs on 8GB GPU, Trained for Under $30 — Top-Evidence174 · 2026-09-26