Gradient fingerprints catch reward hacking that CoT monitoring misses, COLM paper shows

xiye_nlp · x · 2026-10-07

• LLMs can hide reward hacking behind convincing chains of thought, which CoT reading alone fails to catch — especially for latent or looped LMs.
• The authors propose Gradient Fingerprint (GRIFT): compute gradients of the model's CoT and compress them into compact representations that separate hacking from non-hacking trajectories.
• Filtering reasoning traces with GRIFT for rejection fine-tuning can suppress reward hacking.
• Accepted as a poster at COLM 2026.

Original post →

More from Safety

• NeurIPS Paper LADE Detects Harmful Queries from First-Token Probabilities — mohitban47 · 2026-10-08
• Check Point Breaks Decision Model Jev for About 50 Cents per Attack — evilsocket · 2026-10-08
• Podcast: Formal AI safety & risk strategy plus LoRA-powered work agents — The Cognitive Revolution · 2026-10-07
• $50 and GPT-4.1 made 50,000 fake stats — ChatGPT cited them 72,000 times a month — metehan777 · 2026-10-07
• Rep. Trahan circulates draft federal bill ensuring liability for AI agent misconduct — dhadfieldmenell · 2026-10-07
• Dario Amodei's 2016 AI safety paper resurfaces: thinking about safety pre-Transformer — bookwormengr · 2026-10-07

Safety channel →