llama.cpp Adds Decision Models: One-Forward-Pass Option Scoring in as Fast as 3ms
kalyan_kpl · x · 2026-10-03
llama.cpp server now supports decision models via a /v1/systemone endpoint: send a state (text, JSON, screenshot) plus typed questions, and the model returns per-option probabilities in a single forward pass—no text generation or parsing. Supported models include Julia-1 (144M, 3ms), Laya, Kev-4B, lev, and OpenJev (27B, multilingual with image support). Typical uses: request routing, moderation, agent step verification.
More from coding & agent
- Popular Grok Bot 'Chief of Staff' template: quiet, budget-conscious program management — RachelVT42 · 2026-10-03
- Visual tool for managing nested agents: chat threads break at scale, says LukeW demo — Wattenberger · 2026-10-03
- Vibe-coded browser CS 1.6 is fully playable with up to 16 players — TAbrodi · 2026-10-03
- Claude Code 2.1.288 ships 89 CLI changes, adds OAuth re-auth and draft recovery — ClaudeCodeLog · 2026-10-03
- Survey: 42% of devs say AI writes at least half their code, saving 13 hours weekly — rseroter · 2026-10-03
- Most Claude Setups Get Worse as You Add More: Why 90% of Your Hacks Do Nothing — alex_verem · 2026-10-03