SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs
Arnon Mazza, Elad Levi
cs.AI, cs.CL
2026-10-22
SCOPE compiles policies into neural predicates plus Python composition code, letting cheap non-reasoning models match frontier judges at 9-47x lower inference cost.
Policy enforcement for LLM agents has outgrown single-turn moderation. The unit of judgment is now a multi-turn transcript or a full tool-call trajectory, and the policy is rarely one flat category: it bundles conditions, exceptions, counts, and interactions between clauses. Both standard routes break. Fine-tuned guardrail models are fast and cheap, but their capacity is the ceiling, and accuracy slides once a policy compounds enough interacting conditions. LLM-as-a-judge, where a frontier model reads the policy at runtime, is the most expressive option but pays a frontier call on every gated interaction and returns unstable verdicts, with variance rising as clauses accumulate. Neurosymbolic methods, which split the policy into small judgments composed under explicit logic, look like the right shape, but existing symbolic layers are either too inexpressive (checklists and weighted rule sets cannot hold conditional branching or runtime counting) or are fit to a fixed dataset rather than to the policy itself.
SCOPE (Symbolic Composition over Predicates and Extractors, from Plurai) compiles a natural-language policy into an executable program built from three components:
Making the composition layer code rather than a rule formalism is where the expressiveness comes from. Structures real policies actually carry, such as asking the next question depending on the previous answer, repeating a check over runtime-selected spans, or exact counts, are a few lines of Python. Predicates are not bound to an input; the composition code decides what each one receives, so the same predicate can run on the full input, on a span, or repeatedly within one sample.
The program and its training data are optimized jointly. The search starts from the policy text itself as the single predicate. Each round, the optimizer executes the program on the training pool and uses execution traces to attribute every failure to a predicate's wording or to the composition logic. Two adversarial generators then produce counterexamples: a systemic one targets clauses the composition omits or misroutes, a local one targets a single predicate whose wording drifts from its clause. A strong teacher labels each counterexample under the full policy, and it enters the pool only if that label matches the generator's intent. The loop borrows from counterexample-guided synthesis, with the policy as the specification.
The executor is deliberately weak. Predicates are answered during search by a sub-frontier model with reasoning off, so any predicate it cannot handle gets reworded or split; weaker executors yield more predicates. The strong-model bill is paid once, at compile time.
Four policy tasks: auditing an automated critic (Coaching), counting cross-turn intent repetition to judge escalations (Escalation), checking plan validity inside a 45k-character agent trace (Plan), and multi-label tile reference (Tile-match). With the weakest executor, gpt-5.4-nano:
| Method | Coaching | Escalation | Plan | Tile-match |
| Direct prompting | 0.60 | 0.56 | 0.59 | 0.38 |
| Prompt opt (best of GEPA/MIPROv2) | 0.69 | 0.61 | 0.66 | 0.56 |
| AFlow | 0.83 | 0.86 | 0.75 | 0.68 |
| SCOPE | 0.85 | 0.82 | 0.87 | 0.78 |
SCOPE takes or ties the best score in 10 of the 12 task-by-executor cells; AFlow wins the other two, both on Escalation with the weaker executors. Decomposition also compresses the capability hierarchy: moving from the weakest to the strongest executor buys 0.26 accuracy under prompt optimization but only 0.09 under SCOPE. Programs compiled with nano transfer as-is to five open-weight models with no further optimization, averaging 0.13-0.22 gain per model; DeepSeek-V4.1-Flash and Qwen3-235B land in the same range as gpt-5.5 with high reasoning and gemini-3.1-pro judging the tasks directly, and DeepSeek posts the top score of any model on two tasks. Estimated inference cost runs $0.23-1.49 per 1,000 samples against $6.88-50.79 for gemini-3.1-pro, a 9-47x gap. On stability, five runs of Tile-match flipped at least one verdict for only 5-7% of samples under DeepSeek/Qwen3-235B versus 37% for gpt-5.5. The ablation confirms the adversarial loop: removing it drops Plan from 0.865 to 0.813 (p=0.005) and Coaching from 0.839 to 0.783 (p=0.033), with no effect on Escalation.
Teams building guardrails, moderation, or compliance checks get a production-shaped path: recompile when the policy changes, run everything on cheap non-reasoning models, and trace every verdict to the predicate outputs behind it. Compared with fine-tuned guardrails, there is no annotation campaign to run; compared with LLM-as-a-judge, there is no frontier bill per call and far steadier output. Cross-executor transfer is the most practical property: compile against the cheapest API model, deploy on self-hosted open weights, with no further optimization in between.
The authors state the main one themselves: the program is checked by a strong teacher's (Claude Opus 4.8) reading of the policy, not by a formal verifier, so correctness is capped by that reading, and verifying such programs remains open. Coaching and Tile-match are author-built datasets of roughly one to two hundred samples each. CheckEval and RLIE support binary labels only and ran on two of the four tasks. AFlow was allowed to elicit reasoning in intermediate steps, a capability the other methods lack; the paper discloses this, but the comparison is not clean. The Escalation ablation shows no effect from adversarial samples, so that task's gains come from decomposition alone. And the compilation cost itself, a strong-model agent loop of up to eight rounds, is never reported; only inference-side costs are.