llama.cpp adds decision models: score options in one forward pass, from 144M Julia-1 at 3ms
ggerganov · x · 2026-10-02
llama.cpp now supports decision models via a new /v1/systemone endpoint, following the System One format from TypeSafe's Jev model.
Unlike chat models that generate tokens one at a time, a decision model reads a state (text, JSON, or a screenshot) once and returns a probability for each option you provide in a single forward pass. Typical uses include request routing, content moderation, verifying agent steps, and choosing an agent's next action.
Supported models:
- Julia-1 (144M, mmBERT-small based, 50+ languages, Apache 2.0, median 3ms per question)
- Laya (421M, ModernBERT-large, English, 5ms)
- Kev-4B (4B, Qwen3.5-4B-Base, 12ms)
- lev (4B, Qwen3.5-4B, 36ms)
- OpenJev (27B, multilingual incl. Chinese, image-capable, CC BY-NC 4.0, 43ms)
Benchmarks were run on one NVIDIA RTX PRO 6000. The API supports choice and score question types. Quick start: llama serve -hf ggml-org/Kev-4B-GGUF.
More from coding & agent
- Dev recaps Factory podcast: accepting AI output as-is gives no competitive edge — Madisonkanna · 2026-11-17
- Factory dev: stop treating AI tech debt as a 'future model problem' — GPT-6 still can't fix it — Madisonkanna · 2026-11-17
- Notable AI builder says workflow has shifted to cloud agents with unlimited Claude tokens — jiayuan_jy · 2026-11-17
- Jane Street's yminsky: agents are bad at software design — but does design still matter? — charliermarsh · 2026-11-17
- Linear: by October, 53% of PRs were reviewed by an agent and the author alone — jiayuan_jy · 2026-11-17
- Five-stage agent routine that stopped an LLM deleting my repo — with the Brief prompt to steal — Iiifebri · 2026-11-17