A local agent that writes no text: zero invented values in 10,888 calls

System 1 agents: a method for local, reliable and fast agentic AI

Eleftherios Tzanis, Michail E. Klontzas

cs.AI

2026-11-16

A compiler turns schemas and requests into multiple-choice items; a local model only picks. Best agent: 97.0% tool choice on 1,952 requests, zero invented values in 10,888 calls.

What problem this solves

A clinical agent still writes each tool call token by token. A cloud model sends the request, and the patient data it points at, off site. A local model keeps the data in the building and still types values the user never gave: a missing tool, an argument the schema does not have, a hyperparameter or a path. Later steps inherit that result.

Most of what a correct call needs is already in the request, follows from a fixed rule such as a unit conversion, or is listed in the schema. The rest is a choice. A System 1 agent makes only those choices. It runs locally, scores a finite option list in one forward pass, and generates no text.

Method

The agent runs the workflow, a compiler writes the questions, and a decision model answers them. Code then executes the chosen option as written.

Tool selection is one question over every catalogue tool plus none of these. If the top two probabilities differ by at most 0.3, a follow-up asks which tool's parameters match the values in the request. A yes/no question asks whether that tool can do the job, and the agent declines below 0.5. A missing required argument is never guessed. The user gets a fixed prompt from the argument name and the first 120 characters of its schema text. The evaluation had no user in the loop, so missing arguments were listed in the plan. A product of step probabilities below 0.3 also forces a decline, but only in deployment. That rule was off during scoring.

The compiler is deterministic code. Enumerations are asked over their legal values, booleans over on, off, or not mentioned. Paths, numbers, and free text are chosen only from the request, fixed derivations (unit conversions, numbers written as words, paths joined from written parts), the schema, and earlier steps, plus not specified. List items are picked one at a time, and a used value cannot be picked again. The first list item, an optional argument, and a value claimed by two arguments all need probability 0.7. A value equal to the documented default is dropped. A required value the request never states is reported missing. An answer below 0.5 is marked uncertain.

Six published decision models, 4B to 35B, run without fine-tuning through llama.cpp on one GPU, one forward pass and no sampling. Decider reads log-probabilities of the option labels. pplx-decider uses non-causal attention and a readout matrix on the marker state. OpenJev and Clef use the systemone endpoint. OpenJev splits questions above 52 options into groups and multiplies the probabilities. The same build returns the same distribution. Answers are stored by hash and can be replayed question by question.

The compiler was changed only on development requests. An oracle that knows the label picks it whenever the option is offered, which separates a missing option from a wrong pick. Only missing options were fixed, and only with general rules about how requests write values. A change had to raise the total of fully correct calls across models and stay inside a tolerance on earlier reference requests. Code and config were hashed before the test run.

The catalogue has 157 tools on 18 MCP servers. Imaging contributes 42, including nnU-Net, Merlin's CT foundation model, PyRadiomics, TotalSegmentator, and PyCaret. Biomedical tools contribute 115, including 53 proteomics tools, 22 from AlphaGenome, plus Scanpy, ClinicalTrials.gov, and ChEMBL.

Results

The main benchmark has 1,952 requests: 1,645 complete ones stating 6,056 argument values, and 307 out of scope. Another 174 underspecified requests were scored apart. Of 2,126 requests, 123 were copied from ReclAIm and mAIstro notebooks. The rest were written for the study. Controls are nine open-weight LLMs with reasoning on and off, 18 settings. The request was the only user message, tools were native function calls, nothing was executed, and only the first call was scored. Three of the LLMs are the base models of five decision models. Runs used one RTX PRO 5000 (48 GB).

MethodMetricResult
pplx-decider-27btool / arguments / fully correct97.0% / 97.0% / 93.9% (95% CI 92.7-94.8%)
OpenJevsame96.1% / 96.1% / 92.9%
six System 1 agentssame91.7-97.0% / 89.8-97.0% / 81.4-93.9%
Qwen3.5-27B, reasoning offsame95.1% / 97.2% / 91.3%
10,888 System 1 callsungrounded values0 of 33,945
LLM callsat least one invented value5.5-19.7%
174 underspecified requestsSystem 1 / LLM86.2-93.7% / at most 5.7%
time, energy per fully correctOpenJev / best LLM2.6 s, 0.228 Wh / 4.5 s, 0.413 Wh

On fully correct requests, 108 paired tests with Holm correction give System 1 a significant lead in 73, the LLM a lead in 11, and no difference in 24. All 11 LLM wins are against the two weakest System 1 agents. pplx-decider-27b beats every LLM setting. Tool choice is 76 leads to 5. Argument filling is closer: 46 System 1 leads, 19 LLM leads, 43 ties, and five LLMs are not significantly behind the best System 1 agent. The two pplx-decider agents beat their own base model, run as an agent, by 2.6 to 9.6 points. decider-35b-a3b loses to its base model with reasoning on. System 1 leads more on imaging tools with many technical arguments. LLMs fill identifiers better on ClinicalTrials.gov and ChEMBL.

System 1 declined 87.0-94.5% of out-of-scope requests and left 1.0-3.3% of complete requests uncalled. LLMs declined 41.4-97.4%. The two that declined most also skipped 24.3% and 36.5% of complete requests. LLMs never called a tool outside the catalogue, but three of nine used argument names missing from the schema, 19 to 53 calls per setting. Reasoning did not remove invented values. The most accurate LLM still invented the missing required value on 27.6% of underspecified requests, and up to 62.1% of those requests were filled with an invention or left without a call.

GPU energy per fully correct request was 0.072-0.407 Wh for System 1, 0.198-2.481 Wh for LLMs with reasoning, and 0.080-0.853 Wh without. A System 1 request is about 12 questions and zero generated tokens. LLMs wrote 123-340 tokens with reasoning off and 417-952 with it on. Latency was 1.2-10.1 s, against 2.6-26.1 s with reasoning and as low as 1.1 s without. Qwen3.5-35B-A3B with reasoning used less energy than OpenJev. Exact matching of search text changed none of 324 pairwise tests. PyCaret's own model-name table flipped one System 1 win into a non-significant gap. Giving every LLM its better reasoning mode, which favors the LLMs, still left System 1 ahead in 33 of 54 pairs and the LLM ahead in 8. On the 123 published prompts, mean tool choice and argument filling for System 1 were at least as high as for the LLMs in either reasoning mode. Replaying stored answers rebuilds all 12,756 System 1 records.

Why it matters

Cloud models move sensitive data off site. Local models keep it there and then let an invented argument travel into the next step. System 1 only picks. The tool name comes from the catalogue, the argument name from the schema, and the value from the request, the schema, or a fixed policy. Each decision has a probability, and the run can be replayed.

The framework is open source. The decision models are not fine-tuned. An MCP tool plugs in with the name, description, and JSON schema the server already publishes. If a required value is absent, the agent stops and asks. The accuracy gain is modest: 93.9% fully correct against 91.3% for the best local LLM, and that LLM is 0.2 points higher on argument filling. The split shows up when a required value is missing, 86.2-93.7% versus at most 5.7%.

Limitations

Wrong choices among grounded options remain. The agent can pick the wrong tool, or place a value that does appear in the request into the wrong argument. In 22 of 10,888 calls the text was in the request but in a form the schema type or pattern rejects. Checking each option against the schema before showing it would remove those 22. That check was not done. Grounding is loose: any string anywhere in the request counts, so hallucination is undercounted. Each System 1 agent also filled a missing required argument with some other value from the request in 2 to 6 cases. If no output folder is named, a fixed policy fills it, and that value is excluded from the invention count.

The hardest out-of-scope slice, for every agent, is a tool that can do the task but not on the given input. Custom prompts dominate, 2,003 against 123 published ones. LLM tools were never executed. Only the first function call was scored. Multi-step runs keep the first 1,200 characters of the previous result, and long chains have no separate score. The 0.3 product threshold used in deployment was off for the benchmark. Compiler rules were fit on development requests. They are general, and the code was hashed before testing, and they can still have learned the habits of these 157 tools. decider-35b-a3b as a System 1 agent lost to its base LLM with reasoning on, so a decision model is not automatically a gain.

Terms

Source

Related papers

All paper explainers