llama.cpp adds decision models via /v1/systemone, returning option probabilities in a single forward pass
ggerganov announced decision model support in the latest llama.cpp builds, exposed through a new /v1/systemone endpoint: given a state (text, JSON, or screenshot) and typed questions, the model returns per-option probabilities in a single forward pass instead of generating tokens that must be parsed. The feature has drawn wide attention in the community.
Confirmed
- Announced by ggerganov himself; the endpoint follows the System One format introduced by TypeSafe's Jev models, with an API compatible with that format.
- Input states can be text, JSON, or screenshots; questions carry type annotations.
- The smallest model is 144M parameters with inference as fast as 3ms; ngxson confirmed 5 open-source models are supported, with more being added.
- Existing clients only need to change the base URL, enabling local, efficient, private Jev-style inference.
- The ggml team published a blog post on Hugging Face, relayed by users such as paf1138 and adnanhashmi.
- solyarisoftware noted the endpoint suits request routing, tool selection, and validating agent outputs.
Why it matters
- Decision models turn generative calls into scoring calls, removing the cost of token-by-token generation and the uncertainty of output parsing for high-frequency structured decisions.
- Millisecond-level local inference (3ms) makes lightweight models viable for real-time and privacy-sensitive use cases.
- Notable developers including mitsuhiko (Flask author Armin Ronacher) shared the news, showing its reach in the developer community.
2026-10-02 ~ 2026-10-03 · 8 related posts
Primary sources
- llama.cpp adds decision models: /v1/systemone scores options in a single forward pass — ggerganov ·
- llama.cpp adds decision models: typed questions, per-option probabilities in one forward pass — ngxson ·
- llama.cpp adds decision models: score options in one forward pass, from 144M Julia-1 at 3ms — ggerganov ·
- llama.cpp adds Decision Models, expanding local inference capabilities — paf1138 · 2026-10-02
- [source] llama.cpp adds decision models: typed questions, per-option probabilities in one forward pass — ngxson · 2026-10-02
- llama.cpp ships /v1/systemone endpoint for local Jev-style decision models — mitsuhiko · 2026-10-03
- llama.cpp adds /v1/systemone endpoint to run decision models locally, returning probabilities — solyarisoftware · 2026-10-03
4 near-duplicate retellings: ggerganov · ggerganov · adnan_hashmi · kalyan_kpl