llama.cpp adds decision models: typed questions, per-option probabilities in one forward pass
ngxson · x · 2026-10-02
llama.cpp server now supports decision models via the /v1/systemone endpoint: send a state (text, JSON, or screenshot) plus typed questions, and get a probability for every option in a single forward pass—no token-by-token generation or output parsing. The API follows the System One format from TypeSafe's Jev model, so existing clients only need a new base URL.
Supported open models (5, from 144M to 27B):
- Julia-1: 144M, mmBERT-small-based, 50+ languages, 3 ms per question
- Laya: 421M, ModernBERT-large-based, English, 5 ms
- Kev-4B / lev: 4B, Qwen3.5-based, 12/36 ms
- OpenJev: 27B, Qwen3.8-27B-based, multilingual (en/de/fr/hi/zh/ja), image input, 43 ms
Typical uses: request routing, content moderation, verifying an agent's step succeeded, or choosing an agent's next action. Question types include choice (options with per-option probabilities) and score. Quick start: llama serve -hf ggml-org/Kev-4B-GGUF. Models are collected on Hugging Face with a community Decision Index for comparisons.
More from coding & agent
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19