Process rewards boost long-horizon agentic RL 45.4% but cause reward hacking on short tasks

philo-labs · hf · 2026-11-05

A University of Pennsylvania / Philo Labs paper uses video editing as a testbed for when fine-grained credit assignment helps multimodal agentic RL.

Four movie-trailer editing task types — Content Selection, Narrative Sequencing, Rough Cut, Fine Cut — mirror real editing workflows across complexity and interaction horizons.
Decaying process rewards yields the largest gain on Fine Cut: a 45.4% relative improvement over outcome-only GRPO on tasks averaging 60+ turns.
On the three shorter-horizon tasks, fine-grained process rewards cause reward hacking, mitigated by decaying their weight; on-policy self-distillation provides little benefit.
Takeaway: the value of process rewards depends on task horizon — they pay off only in long-horizon, delayed-feedback settings.

More from Research