Poor-acting decision models can still teach RL agents well on ALFWorld and WebShop

ZhejiangUniversity · hf · 2026-11-13

Researchers from Zhejiang University and Baidu studied decision models in long-horizon agents, comparing the same model as actor, judge, or teacher:

Decision models can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with RL. The authors ask: can a decision model that acts poorly still teach well?
As actors, all models get pulled away from a subgoal by an easier one. As judges seeing the whole trajectory, the same models credit steps toward that subgoal — asking whether each action helped, rather than whether the task succeeds, separates useful from useless actions.
As a teacher, judging complete action trajectories improves RL policy success on ALFWorld and WebShop, showing weak-acting models can still effectively teach RL policies.

More from coding & agent