llama.cpp Adds Decision Models: One-Forward-Pass Option Scoring in as Fast as 3ms
kalyan_kpl · x · 2026-10-03
llama.cpp server now supports decision models via a /v1/systemone endpoint: send a state (text, JSON, screenshot) plus typed questions, and the model returns per-option probabilities in a single forward pass—no text generation or parsing. Supported models include Julia-1 (144M, 3ms), Laya, Kev-4B, lev, and OpenJev (27B, multilingual with image support). Typical uses: request routing, moderation, agent step verification.
More from coding & agent
- FT: US tech outages blamed on rush to fix bugs found by AI — nordicinst · 2026-11-19
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19