Hacking open-source model behavior with sglang's scoring endpoint, no fine-tuning needed
BLUECOW009 · x · 2026-09-19
Developer skeptrune shows a surprisingly simple inference-engineering trick to make any open-source model behave in a targeted way — demoed with a joke endpoint named deepseek-v4.1-flash-jev.
The core idea: sglang exposes a scoring endpoint (/v1/score) alongside normal generation. In scoring mode, given an input and candidate answers, the model is forced to output a probability distribution over the options instead of streamed text (e.g., "most common letter in abcccde" → (c, 0.9)). Swapping /gen for /v1/score is all it takes.
By engineering over these score distributions, you can cheaply replicate a specific model's behavior style without any fine-tuning. Commenters quip that future models might just ship with a built-in "jev option."
More from Infra
- GPU host warns: renter exploited his rig for attacks, Clore.AI blocked him for reporting it — anomaly256 · 2026-09-19
- 3M paid $10.3B to quit PFAS — AI data centers just made it a growth market again — aakashgupta · 2026-09-19
- $250 of modded mining cards, 30GB VRAM: old i7 PC runs Qwen at 30 tok/s with patched drivers — HFq_Dev · 2026-09-19
- Reading a Pretraining Run: A P0/P1/P2 Metric System for Monitoring LLM Pretraining — SonglinYang4 · 2026-09-19
- Dual RTX 5060 Ti Only Gets 10 t/s on Qwen3.8-Flash-Next, Seeking Config Advice — MkGod · 2026-09-19
- Distilling DeepSeek V4 Flash to a 4B model on DGX Spark: 26 hours, 22ms per judgment — Dan_Jeffries1 · 2026-09-19