ReCouPLe: Reason-Augmented Preference Learning Boosts Reward Accuracy 1.5x Under Shift
burny_tech · x · 2026-09-21
An ICLR26 paper (arXiv:2603.04861) tackles a core flaw of preference-based reward learning: binary feedback like "trajectory A > B" doesn't say why, letting reward models latch onto spurious correlations (e.g., learning "moving left" when the real preference is collision avoidance).
- Method: ReCouPLe treats each natural-language rationale as a projection axis ψ in embedding space, decomposing trajectory representations into parallel and orthogonal components, so preferences are explained by features aligned with the stated reason.
- Reuse: Rationales sharing semantics ("avoids collisions", "completes faster") reuse the same causal direction across tasks, transferring preference knowledge to novel tasks with no extra data or LLM fine-tuning.
- Results: Up to 1.5x better reward accuracy under distribution shift and 2x downstream policy performance on novel tasks. Code released.
More from coding & agent
- Dev to open-source an Obsidian Agent plugin and one-prompt website deployer — vista8 · 2026-09-21
- JEV eval: when state evidence is ambiguous, choice order dominates and IDK fails 0/80 — iamrobotbear · 2026-09-21
- Repowise Indexes Codebases for AI Agents via MCP, Hits 6.7k GitHub Stars — tom_doerr · 2026-09-21
- Reverse-engineering Jev: its architecture, philosophy, and where it falls down — iamrobotbear · 2026-09-21
- Veteran dev: AI built a full MMO in 2 hours, so what am I for? — devontec · 2026-09-21
- Workflow tip: dictate rambling prompts to ChatGPT, then paste into your agents — athyuttamre · 2026-09-21