llama.cpp adds decision models: /v1/systemone scores options in a single forward pass
ggerganov · x · 2026-10-02
llama.cpp server now ships a /v1/systemone endpoint for decision models: you send a state (text, JSON, or a screenshot) plus typed questions, and the model returns a probability for each option in a single forward pass—no token-by-token generation or output parsing. The API follows the System One format introduced with TypeSafe's Jev model, so existing clients only need a new base URL (implementation in PR #29818).
Typical uses: request routing, content moderation, verifying an agent's step succeeded, or picking an agent's next action.
First models (median latency per question on one RTX PRO 6000):
- Julia-1: 144M, based on mmBERT-small, 50+ languages, Apache 2.0, 3ms
- Laya: 421M, ModernBERT-large, English, Apache 2.0, 5ms
- Kev-4B: 4B, Qwen3.5-4B-Base, English, Apache 2.0, 12ms
- lev: 4B, Qwen3.5-4B, English, Apache 2.0, 36ms
- OpenJev: 27B, Qwen3.8-27B, en/de/fr/hi/zh/ja, image support, CC BY-NC 4.0, 43ms
Quick start: llama serve -hf ggml-org/Kev-4B-GGUF. Questions include choice (returns top option plus per-option probabilities) and score types. Models live in the Hugging Face Decision models collection, with a community Decision Index for comparisons and more models on the way.
More from coding & agent
- AI in finance shows a power law: top 1% users generate 35x more files with agents — peterjliu · 2026-11-19
- Linux LTS patch flood: 3318 patches for 6.6 alone, 'no way humans are reviewing this' — pkqzy888 · 2026-11-19
- Caliper Harness: open-source coding harness disciplines small local models with plans, checks, reviews — Saraozte01 · 2026-11-19
- AI systems lack executive function, not scale: four manager-layer strategies to fix it — Designer-Material633 · 2026-11-19
- Google employee replaces doomscrolling with 3 fully automated personal AI newsletters — DynamicWebPaige · 2026-11-19
- Designer spins up an agent to build state-airline brand gallery mid-flight delay — beh_zod · 2026-11-19