CAMEL: Confidence-Gated Reward Model

机器之心 · wechat · 2026-07-12

CAMEL: Reward Modeling with Confidence Gating

This paper introduces CAMEL, transforming reward models from generative judges that "always output long explanations" into a pipeline that "gives a single-token initial verdict first, then decides whether to reflect based on confidence." The core signal isn't a separately trained confidence head, but the log-probability margin between two verdict tokens: a larger margin means the model is highly certain; a smaller margin triggers a reflection review.

Key Methods

Single-Token Initial Verdict: First determines which candidate response, A or B, is better.
Confidence Gating: High-confidence samples are output directly, while low-confidence samples trigger reflection.
Counterfactual Prefix Augmentation: During training, two versions of the same sample are constructed, forcing initial verdicts of A and B respectively. GRPO is then used to train the model to "confirm correct initial verdicts and overturn incorrect ones."

Experimental Results

The authors trained the model based on Qwen3-14B and evaluated it on RewardBench, RM-Bench, and JudgeBench, covering tasks like chat, safety, math, and coding.

CAMEL-Fast: Scores across the three benchmarks are 90.5% / 74.8% / 65.2% respectively.
CAMEL-Reflection: Achieves 92.8% / 84.2% / 71.6%.
Average accuracy reaches 82.9%, a 3.2% improvement over the previous best baseline.
With 14B parameters, it outperforms multiple 70B-level reward models.

Conclusion

The paper emphasizes that the key to reward models isn't indiscriminately lengthening reasoning, but allocating compute budget to genuinely difficult samples. Post-training, the model's confidence distribution shifts to the left overall, indicating it has become more conservative and better at identifying which samples warrant reflection.

Original post →

CAMEL: Confidence-Gated Reward Model

More from Research