Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization
Alireza Modirshanechi, Benjamin Eysenbach, Peter Dayan, Eric Schulz
NeurIPS 2026 (Oral Presentation
cs.LG, cs.AI, eess.SY
2026-05-07
Three goal rewards can pick incompatible optimal policies in one environment. Skill mutual information bounds goal sensitivity only when the behavioral variable matches.
Unsupervised pretraining is already standard in goal-conditioned reinforcement learning (GCRL): one policy has to reach whatever goal it is given. Mutual information skill learning (MISL) skips the task reward and trains skills until an observer can name the skill from behavior alone. The learned skills are then reused for goal reaching. The open question is why a skill that is easy to identify should also be good at arriving.
Both labels cover families. Goal reaching is scored in at least three ways. Diversity can mean different endpoints, different occupancies, or different full trajectories. The paper reads both families as control maximization. A good goal-conditioned policy is one whose future trajectory moves with the command. The exact meaning of sensitivity depends on the score.
With sparse rewards there are three formulations.
Persistent goal, Pe(γ). The goal stays put. Every visit pays 1−γ, and later rewards are discounted by γ. The return equals discounted state occupancy: sample a future time from a geometric clock and ask whether the state is the goal. C-Learning, contrastive RL, LEXA, and GoFAR use the stationary structure or that occupancy identity. Without real changes they do not apply to finite horizons.
Exact timing, ET(K). The agent scores only by standing on the goal at step K. The simplest HER relabeling replaces the command with the trajectory endpoint, which lines up with this score.
Opportunity window, OW(K, γ). Any arrival within K steps scores, and γ pays for arriving sooner. This covers stochastic shortest paths. Goal GAN proposes goals by the chance of arriving inside the window, so it lines up with OW.
Proposition 4.1: one environment can give the three scores three different optimal policies. The split appears when speed fights reliability. In the river example a jump from s1 succeeds 8% of the time, and the bridge is slower. The appendix takes K=2 and shortcut rates ε1<ε2. When γ is larger than both √ε1 and √ε2 and smaller than ε1/ε2, the three optimal paths differ.
They agree in special cases. A one-step reward makes the three formulations the same problem. An infinite window makes OW and Pe share a stationary optimum, so a policy trained to revisit a goal can also be used to arrive quickly. In a deterministic environment both are shortest paths. Window length and persistence drop out. If every state has a wait action and γ=1, arrive-and-wait is optimal for both finite-horizon scores.
Goal sensitivity C is the extra average return from the correct command over a random command. A consistent policy scores better on the goal it is actually pursuing. If the environment cannot be steered, C is 0.
Theorem 5.1: on Pe and ET, J = C + 1/Ns, where Ns is the number of states. Maximal control is optimality. For nonnegative bounded returns, including OW, J ≥ Ns/(Ns−1)·C. On OW a maximally sensitive policy need not be optimal. Regret is at most 1 − Ns/(Ns−1)·C.
Once a deterministic map sends goals to skills, Theorem 6.1 approximates C by mutual information between the goal and behavior. Pe uses discounted occupancy and ET uses the endpoint. Both bounds are attainable and rise with C. The upper bound needs consistency. The lower bound does not. OW calls for the discounted first-visit vector: each entry is the discounted reward for hitting that state for the first time. Mutual information with the whole trajectory is only a loose upper bound. In general OW gives just I(G;F) ≥ 2C².
Proposition 6.1 handles the prior. MISL assumes uniform skills. The downstream distribution is set by the goals and the map. When the number of skills does not exceed the number N' of behavioral outcomes, a total-variation gap δ moves the mutual information by at most h(δ)+δ log(N'²(N'−1)), with h the binary entropy. Uniform goals and equal-sized blocks make the gap 0.
The paper reports no benchmark success rate. What can be checked are the relations above, and a mismatch that scores zero.
| Downstream score | Matching variable | Link to sensitivity |
| Pe(γ), repeated occupancy | Discounted occupancy | MI bounds attainable and increasing in C |
| ET(K), arrive at a set step | Endpoint | Same; in deterministic environments empowerment = log(1+Ns C) |
| OW(K, γ), arrive inside the window | First-visit vector | In general only I≥2C²; trajectory MI is a loose upper bound |
Empowerment is the mutual information between actions and a future state.
Skills can take distinct trajectories and share one endpoint. Trajectory mutual information is then maxed out, and ET sensitivity is still 0 no matter how goals map to skills. An appendix environment with K=2 and γ=1 goes the other way: the higher-return policy has lower sensitivity. On opportunity windows, maximizing control is only approximately enough.
Pick the pretraining objective from the test score. Repeated occupancy matches discounted occupancy. Arrival at a set step matches the endpoint. Arrival before a deadline has no tight off-the-shelf objective.
The uniformity gap δ is a diagnostic. Extra skills do little if the test-time map rarely selects them. A larger skill set does not by itself improve goal reaching.
Deterministic tasks are the exception: shortest paths cover both persistent goals and opportunity windows. A method built for one score should not be assumed to solve another.
There is no new training algorithm. The paper is a NeurIPS 2026 theory result, and the correspondence can be checked from the bounds.
States, actions, and skills are finite. Common benchmarks are continuous, and the mutual-information bounds are the part that is hard to move. Goals are treated as full states. Practical goals often fix position and ignore velocity, so diversity would have to be scored on goal-relevant variables. That extension is not done.
The map is deterministic. A stochastic map can be charged the entropy of skill choice. The uniform-goal assumption is relaxed in the appendix. Consistency is not a global premise. Sections 4 and 5, Proposition 6.1, and the mutual-information lower bounds hold for arbitrary policies. The upper bounds use consistency.
For opportunity windows the general bound points one way, and the best skill objective is still open. If the downstream task is a window, or a variant of one, an existing pretraining objective should not be treated as a guarantee.
On deterministic benchmarks the scores often share a shortest path, and the paper never measures how large the conflict is. The bounds are monotone relations. Variational approximations and initialization can pull a matched objective away from the policy that training produces. That width is not estimated.