Think Before You Act: Decision Gate for LLM Agents
Iftach Shoham, Yuval David, Offer Akrabi, Erez Bilgory, Ido Levy, Sami Marreed, Sergey Zeltyn, Segev Shlomov
cs.AI
2026-10-28
JevAct gates writes and hand-offs with code checks and Kev probabilities. All eight agents raise Telecom pass^3; Opus 4.6 goes from 39.5% to 80.7%. Airline is mixed.
A booking, a record edit, or a cancellation changes the environment as soon as the tool runs. A later apology does not roll it back. Transferring the user to a person, or saying the job cannot be done, ends the attempt just as firmly, often before troubleshooting is finished. ReAct mixes reasoning with tool calls. Self-Refine and Reflexion critique a proposal after the fact so the next attempt can be better. None of that checks a domain policy in the moment before a consequential action is committed.
A second model can write a free-form review, but something still has to parse that review before a gate can act, and doing the review with a frontier model on every call is expensive. JevAct inserts a check only before writes and hand-offs. The original agent still proposes the action. The checker does not choose it.
Offline, Claude Opus 4.6 turns each domain policy and the tool schemas into yes/no questions, a Boolean rule for when a call is allowed, Python that reads tool records and computes facts, and a hand-off checklist. One compile instruction carries a lot of weight: a condition nobody discussed is not treated as already ruled out. Without it, an enable-roaming rule collapses into a clause about the user not traveling, and the checklist never asks whether the user is on the road. Questions that the judge confidently marks as violations on more than 10% of a tool's development calls are dropped. Hand-off steps flagged on more than 5% of hand-offs in solved training episodes are dropped too.
Online, each step is propose, trigger, check, then commit or revise. Messages that neither write nor hand off run as in ReAct. The write gate allows two revisions. The hand-off check allows one. A flagged proposal is discarded. It is not executed and it is not stored. Feedback is attached for that one retry only. When the budget is spent, the next proposal is committed with no further check, so the gate cannot stall the agent forever.
Dates, amounts, and counts already present in tool records are decided in code. Questions that require reading the conversation go to Kev-27B, a decision-model fine-tune of Qwen3.8-27B used with no further training. It returns a probability for each fixed option in one forward pass and does not generate text. Questions on the same prompt share that pass. On the write gate an answer counts as no only when P(yes) is at most 0.05. Otherwise it counts as yes. A code violation or a failed Boolean rule sends the call back, and the feedback names the failed items. After a second flag, the feedback tells the agent to answer the user instead of calling the tool again.
Before a hand-off, a keyword filter catches likely give-up messages and Kev confirms them. A checklist step stays open when the probability that the user's problem needs it is at least 0.5 and the probability that it is already settled is at most 0.5. If the problem is unfixed and a required step is open, the check returns those steps and allows one retry.
BFCL has no written policy, and its user turns are scripted, so JevAct uses two generic checks. If the agent ends a turn while a tool error still says the call needs to happen or must happen, and Kev assigns at least 0.6 probability to one tool that would satisfy it, the feedback says to call that tool instead of asking the user. Before a state change, if the probability that the user or a tool error requested the call is at most 0.05, the call is sent back as unrequested.
The baseline is ReAct with the same agent model, three runs per task. pass@3 is the share of tasks solved at least once. pass^3 is the share solved in all three runs. On τ²-bench the user simulator is GPT-4.1. Every main JevAct run uses Kev-27B as is.
Telecom has 114 tasks. All eight agents improve on both metrics. Mean pass@3 rises by 16.0 percentage points and mean pass^3 by 16.9. Opus 4.6 moves from 39.5% to 80.7% on pass^3, and from 72.8% to 100% on pass@3. GPT-4.1's pass@3 goes from 45.6% to 83.3%, while its pass^3 only goes from 11.4% to 17.5%: more tasks are solved somewhere in the three runs, and few are solved every time. Opus 5.5, the strongest ReAct baseline, still moves from 74.6% to 82.5% on pass^3. These numbers describe the full harness. They do not isolate the hand-off check.
| Model | ReAct | JevAct |
| Opus 4.6 | 39.5% | 80.7% |
| Qwen3.8-27B | 57.9% | 81.6% |
| gpt-oss-120b | 28.9% | 51.8% |
| GPT-4.1 | 11.4% | 17.5% |
| Opus 5.5 | 74.6% | 82.5% |
Telecom pass^3.
Airline has 50 tasks, so one task is 2 points. pass@3 rises for four of six agents, stays at 90.0% for Qwen3.8-27B, and falls from 68.0% to 66.0% for gpt-oss-120b. pass^3 rises for four and falls for two: gpt-oss-20b from 48.0% to 44.0%, DeepSeek-V4-Pro from 68.0% to 66.0%. The clearer gains are GPT-4.1, pass@3 from 66.0% to 82.0%, and Opus 4.6, from 84.0% to 92.0%. Strong agents are rarely stopped. Opus 4.6 averages 0.08 returned drafts per run, and Qwen3.8-27B averages 0.03.
On BFCL multi-turn, 800 entries, pass^3 rises for all nine agents, by 0.1 to 9.6 points. GPT-4.1 goes from 38.2% to 47.8%. Gemma 4 31B goes from 61.5% to 61.6%. pass@3 rises for four agents and falls for five, by 0.2 to 3.6 points. All-run success can improve while the set of tasks ever solved does not grow.
A single-trial Airline ablation with GPT-4.1 splits the gain. ReAct pass^1 is 50.0%, the full write gate 72.0%, code checks alone 68.0%, and Kev questions alone 57.1%. Flagging every write once, with no validity check, stays at 50.0%. With three runs and a gpt-oss-120b user simulator, that unconditional recheck moves GPT-4.1 pass^3 only from 26.0% to 28.0%, while the full gate reaches 40.0% (paired 95% CI +2.0 to +26.0). For gpt-oss-120b as the agent, the same recheck, or random flags at the gate's rate, reaches 42.0% from 32.0%, matching the full gate. Whether the verdict matters, beyond the extra attempt, depends on which model is proposing.
Other judges also help. On the Airline write gate, pass^1 rises 9.4 to 18.7 points over the matched ReAct baseline. With GPT-4.1 as the agent, Kev-27B scores 72.0% and generative judges land between 67.3% and 72.0%. Estimated judge cost is $0.0003 to $0.0017 per task for Kev, $0.0020 to $0.0040 for open-weight generative models, and $0.0127 to $0.0231 for API judges. Agent inference is excluded.
A larger judge is not uniformly better. On Telecom with Opus 4.6, pass^3 is 97.4% with Kev-9B, 90.4% with Kev-0.8B, and 80.7% with Kev-27B. On 609 identical hand-off states, Kev-9B agrees with Kev-27B on 97.0% of decisions and Kev-0.8B on 90.6%. Thresholds were set for Kev-27B, and the agent still has to act on the feedback, so this is not a ranking of judge accuracy. On Airline, 41% of write-check inputs exceed 8,192 tokens, past the context length validated for the smaller judges, which then miss more invalid calls.
Agent weights, the base prompt, and the tools stay as they are. The gate wraps an existing ReAct loop. On troubleshooting tasks the common failure is stopping early, and all eight Telecom agents benefit. On Airline, code checks alone move GPT-4.1 from 50.0% to 68.0%, and Kev adds the rest of the way to 72.0%. Facts that can be asserted in code do not need a model. A decision model prices the whole question list as one forward pass, and its estimated judge bill sits about an order of magnitude under API generative judges, at similar task success.
This is an incremental runtime guard. pass@3 on Airline and BFCL moves in both directions. It fits tool agents whose policy can be compiled, where a bad write or an early hand-off is expensive, and where one or two retries are acceptable.
Two limits are stated in the paper. With a gpt-oss-120b user simulator, the Telecom gain disappears: pass^3 changes by -1.8 points, 95% CI -7.9 to +4.4. The main Telecom result uses the GPT-4.1 simulator. On 260 labelled development write calls, 42 of 101 incorrect calls violate no compiled question, because the questions encode policy clauses rather than the user's request.
When gpt-oss-120b is the agent, random flags match the full gate at 42.0% pass^3, so the content of the verdict can amount to another chance to answer. The Airline component ablation is a single trial. The hand-off check flagged 26% of hand-offs in solved Airline episodes offline, so the main Airline runs turn it off, and the Telecom jump does not transfer to booking. Thresholds and the dropped-question list were set with Kev-27B. End-to-end latency and safety outcomes are not reported.