PoEM: predicting RL policy outcomes by composing models trained on other rewards

gowthami_s · x · 2026-11-21

A new paper introduces PoEM, a method that predicts the policy RL would learn for a new reward without actually running RL training. Key finding: even when reward functions aren't linearly connected, RL tends to produce log-policies that are approximately linearly related, so a new RL run can be approximated by composing models previously trained on a broad family of other rewards. The authors validate PoEM on both LLMs and diffusion models, with programmatic and neural rewards; qualitative comparisons show PoEM outputs closely matching real RL targets on image rewards like red fox, bald eagle, and lion.

Related event: MIT CSAIL's PoEM Predicts RL Policies on New Rewards Without Training(2 posts)→

Original post →

More from Research

Research channel →