Stanford's Finn lab proposes Generative Value Classifier, recasting Q-learning via action likelihoods

chelseabfinn · x · 2026-10-25

Chelsea Finn's group (Alexander Swerdlow, Perry Dong) published the Generative Value Classifier (GVC) paper:

Core inversion: conventional robot RL critics train from scratch on sparse rewards, ignoring supervision in the policy's pretraining actions. GVC instead asks "how likely is this action under each possible return?", recasting value learning as generative classification.
Method: learn a return-conditioned action model p(a|s,z) and return prior p(z|s); recover the return distribution per bin via Bayes' rule to form an implicit Q-function, with a squared-error loss anchoring Q to the bootstrapped target.
Benefits: critic shares the actor's architecture and every pretraining action supervises learning; gains over leading discriminative methods (incl. distributional RL) on offline and online RL tasks.
Outlook: the authors argue GVC combines gracefully with large pre-trained policies and may generalize better from richer output signals.

More from Embodied