CAMEL: Confidence-Gated Reward Model
机器之心 · wechat · 2026-07-12
CAMEL: Reward Modeling with Confidence Gating
This paper introduces CAMEL, transforming reward models from generative judges that "always output long explanations" into a pipeline that "gives a single-token initial verdict first, then decides whether to reflect based on confidence." The core signal isn't a separately trained confidence head, but the log-probability margin between two verdict tokens: a larger margin means the model is highly certain; a smaller margin triggers a reflection review.
Key Methods
- Single-Token Initial Verdict: First determines which candidate response, A or B, is better.
- Confidence Gating: High-confidence samples are output directly, while low-confidence samples trigger reflection.
- Counterfactual Prefix Augmentation: During training, two versions of the same sample are constructed, forcing initial verdicts of A and B respectively. GRPO is then used to train the model to "confirm correct initial verdicts and overturn incorrect ones."
Experimental Results
The authors trained the model based on Qwen3-14B and evaluated it on RewardBench, RM-Bench, and JudgeBench, covering tasks like chat, safety, math, and coding.
- CAMEL-Fast: Scores across the three benchmarks are 90.5% / 74.8% / 65.2% respectively.
- CAMEL-Reflection: Achieves 92.8% / 84.2% / 71.6%.
- Average accuracy reaches 82.9%, a 3.2% improvement over the previous best baseline.
- With 14B parameters, it outperforms multiple 70B-level reward models.
Conclusion
The paper emphasizes that the key to reward models isn't indiscriminately lengthening reasoning, but allocating compute budget to genuinely difficult samples. Post-training, the model's confidence distribution shifts to the left overall, indicating it has become more conservative and better at identifying which samples warrant reflection.
More from Research
- Nat Lambert shares a reading list on synthetic data and agentic SFT data — natolambert · 2026-07-22
- Turning Noise into Signal: Predicting TCR Binding Using AlphaFold3 Hallucinations — quaidmorris · 2026-07-22
- Lightwheel AI Launches SimReadyGen: Text-to-Physics-Accurate Robot Sim Assets — ZeYanjie · 2026-07-22
- PNAS special issue examines copyright, governance, and AI in the legal system — chrmanning · 2026-07-22
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22