A Constitution-Grid Instrument for Data-Efficient RL Alignment (C-Guard)
Xianling Zhang
COLM 2026 ER
cs.CL, cs.LG
2026-08-01
A constitution grid aims guard RL data. Over-refusal falls 22.4% to 12.8%; 187 untargeted rows bought zero. Adversarial under-refusal rises 0.27 to 0.33 off-scoreboard.
A safety guard has two jobs that pull against each other: block real harm, and let through prompts that only look like harm. The 4B Nemotron-Content-Safety-Reasoning baseline flags "How do I kill someone in Call of Duty?" as unsafe because it sees the word kill. Its over-refusal on benign prompts is 22.4%. The other failure is still there: a harmful request dressed as fiction or roleplay walks through.
The usual fix is more boundary data. The counterexample here is blunt. Adding 187 untargeted rows to the weakest privacy topic moved accuracy by 0. Watching only XSTest, an over-refusal suite, is worse: the other axis can rot while the scoreboard barely twitches. The question is where each new row should land, and whether that region is worth training, before any GPU time is spent.
C-Guard turns a constitution (one policy clause per harm topic) into a grid. Rows are topics, columns are ways of asking. At each clause boundary the generator writes a twin pair that shares a trigger word and flips only intent: one should pass, one should be blocked. The guard has to learn the boundary, not the word.
C-LIM reads the board. Each cell is sampled 8 times on rows the model has never trained on, and the pass-rate trajectory is scored. A cell that climbs with the field mean scores near 1: mastered, prune it, keep old rows as a retention set. A cell that stays flat under a rising field can score a large negative: dead weight. LIMR scores training samples and then subsets them. C-LIM scores a region, on probes that never enter training.
Moves follow the read:
Every cell is read from both sides. The benign channel checks that safe twins pass. The attack channel checks that disguised unsafe prompts in the same style are blocked. XSTest stays outside the loop as a read-only scoreboard, which is how training drift gets measured on data the model never saw.
Training starts from SFT-only Nemotron-Content-Safety-Reasoning-4B. The recipe is vanilla GRPO: 1 if the answer slot matches the label, minus 0.2 for invalid format, including a leaked label in the reasoning trace. Rollout groups that all agree carry zero gradient and are dropped online, so compute sits on prompts the model is still inconsistent about. The corpus grows from about 2K rows to 7,241.
C-LIM marks the dead zone before the budget is spent. Twenty of 21 scored cells look healthy. privacy|public-vs-private stays at 0.80 while the field climbs to 0.98, C-LIM -11.9 against -0.33 for the next-worst cell. Those 187 untargeted rows moved accuracy by 0.000. Once rows were aimed at the probed failures (personal facts about fictional characters, protected attributes of real acquaintances), learning impact on that region rose from 0.733 to 0.80.
Against the SFT base, the shipped model looks strong on XSTest: over-refusal 22.4% to 12.8%, balanced accuracy 0.876 to 0.921, pair consistency 0.690 to 0.810, under-refusal on the scoreboard 2.5% to 3.0%. Independent sets tell the other half.
| Eval | Over-refusal base→ship | Under-refusal base→ship |
| XSTest (scoreboard, 450) | 0.224→0.128 | 0.025→0.030 |
| WildGuardTest adversarial | 0.108→0.062 | 0.267→0.328 |
| WildGuardTest vanilla | 0.114→0.043 | 0.128→0.155 |
| ToxicChat real traffic (2,853) | 0.059→0.042 | 0.144→0.210 |
Over-refusal falls on every eval. Under-refusal barely moves on the scoreboard and rises on all three independent slices, worst on WildGuardTest adversarial, 0.267 to 0.328. The paper calls this the drift tax: the boundary shifted, it did not sharpen. ToxicChat is human-labeled live traffic with the same shape, so the tax is unlikely to be an artifact of the generator.
The gates blocked real moves. One amendment flipped 4 of 60 settled precedents and was rejected. A new topic, sexual content, learned itself and dropped global balanced accuracy from 0.944 to 0.916, with privacy and discrimination transferring down, so the benefit gate sent it back.
For people doing guard RL, the portable piece is the measurement, not the 9.6-point over-refusal drop. XSTest has 450 prompts and a median length of 8 words. Used as the only scoreboard it will congratulate you. A second channel plus an independent attack set shows when the training target has drifted. C-LIM flags dead-weight data before training, which is cheaper than train-then-validate.
The authors shrink their own claim: the loop and the two measurements, not a win over volume. There is still no evidence that aimed coverage lowers both axes at once. Classification is 9.0% of OpenRouter spend and 6.9% of tokens, so guards are a real workload. This paper is about aiming data at two objectives that fight, not a new SOTA table.
The same instrument is proposed for other conflicting pairs: helpfulness versus harmlessness in a chat model, precision versus recall in a retriever, brevity versus completeness in a reasoner. Those runs are not in the paper.
RL moved the guard along the over-refusal / under-refusal frontier rather than past it. Aimed rows helped on the helpfulness axis for the privacy cell. The authors say they do not yet have the matching evidence on the harmlessness axis. Closing the drift tax is listed as future work.
The base is a 4B reasoning guard whose errors lean toward over-refusal, the side with headroom. A verdict-only classifier has no rollout variance, so this GRPO recipe does not transfer cleanly. This is a single-author independent study. The main comparison is the model's own SFT start, not a shared protocol against WildGuard or Llama Guard on post-RL under-refusal. XSTest is short and trigger-matched, unlike production traffic. The independent sets patch that; they are still not an online A/B.
The zero-gain 187-row story and the 0.733 to 0.80 lift both come from one privacy family. The narrative is complete. Intervals are not.