PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM Agents
Seongjae Kang, Taehyung Yu, Sung Ju Hwang
cs.AI, cs.CL, cs.LG
2026-08-20
PolicyGuide compiles policy into a workflow graph and guides agents at user turns. Mean Pass4 rises from 0.42 to 0.62, telecom 0.19 to 0.61, lowest attack-success rate.
Customer-service LLM agents change flights, orders, and plans on a user’s behalf while staying inside company policy. Failures split in two: granting an ineligible change, or skipping identification, confirmation, or diagnosis. Runtime guards usually fire on the final mutating tool call, so diagnose–instruct–verify steps never reach the guard. Workflow and SOP agents can drive a procedure, but they optimize for finishing the flow, not for restraining a general-purpose agent.
In the authors’ source-policy audit, procedural requirements cover 67.4% of airline, nearly 100% of retail, and 98.0% of telecom. Ordered workflows concentrate in telecom (54.0%) against 4.7% airline and 3.6% retail. Flat prerequisites can often be checked at a mutation; ordered requirements constrain earlier dialogue and read-only tools. PolicyGuard already returns remediation at a mutating call and remains action-triggered, so deviations outside that action class slip through.
Offline, the policy and tool registry compile into a frozen workflow graph: a shared intake and identification spine, then request-specific subflows. Nodes carry an explicit satisfying condition; mutating tools unlock only at authorization nodes. Online, an external verifier fires before the agent replies to each user turn, and again if a mutating call is not authorized by current graph state.
The verifier reconciles open requests, walks from persisted positions, and stops at the first unsatisfied node, whose required action becomes the remediation. Facts and eligibility count only from tool results; user choices and consent count from user messages. Code, not chat memory, owns graph position; unknown node IDs are rejected. The evaluated mode is advisory: the first unauthorized mutation in a user-turn region is intercepted, then a one-shot gate disarms so the loop cannot deadlock. GPT 5.4 authors one frozen graph per domain; Claude and Gemini executors reuse it so the comparison isolates runtime, not re-authoring.
Main table: GPT 5.4, n=4. Pass4 is reliability across four trials.
| System | Airline Pass4 | Retail Pass4 | Telecom Pass4 |
| ReAct | 0.460 | 0.596 | 0.193 |
| PolicyGuard | 0.580 | 0.360 | 0.202 |
| PolicyGuide | 0.620 | 0.614 | 0.614 |
Domain-mean Pass4 moves from about 0.42 to 0.62. Telecom Pass1 goes from 0.384 to 0.866 and Pass4 from 0.193 to 0.614, the most workflow-structured domain. On retail, PolicyGuide keeps ReAct’s mutation success while raising policy-violation tasks; PolicyGuard’s violation gain comes with a mutation drop. The overall retail gap versus ReAct is not significant.
Ablations: stuffing the graph into the actor and dropping the external verifier (SELF) does not beat ReAct on mutations. Keeping the verifier but swapping the graph for raw policy (RAW) scores 0.350 Pass4 on telecom versus 0.675 for the full system. FlowAgent, given the same graph compiled to PDL, scores 0.350 on the telecom test split. The airline graph transfers to Claude Sonnet 4.6 (0.780 vs ReAct 0.720) and Gemini 2.5 Pro (0.680 vs 0.480). On 20 CRAFT airline attacks, per-trial ASR is 0.087 against 0.125 for PolicyGuard and 0.200 for ReAct. On an author-designed telecom order audit, process-valid rate is 56.2% versus 17.5% and 13.1%.
The unit of policy adherence is the procedure, not the last tool call. Putting the graph in the actor prompt is not enough; persisted external position and step-level remediation are what move long diagnostic chains from 0.19 to 0.61. For closed models that cannot be finetuned on a natural-language policy, this is a drop-in runtime layer. Verifier calls cost about $0.40 per conversation. Smaller models and sparser firing cut that bill; they do not delete it.
Three English τ²-bench domains and a frozen user simulator do not stand in for other policy regimes, languages, or live users. Retail has only 10 policy-violation tasks. The benchmark scores final state and assertions, not intermediate order; the telecom trace metric is an author-designed operationalization without second-annotator agreement. Node judgments are probabilistic and fail open, so high-stakes actions need another deterministic monitor. Transfer across workflow-author models is untested. Firing at user-turn boundaries leaves gaps between interventions outside any unconditional guarantee.