A note on goal-based hierarchical RL
Kevin Murphy
cs.AI
2026-09-13
A note fuses ACGVF's agent-chosen goals with Murphy's 2025 belief-state policy via the 2001 HHMM, splitting segment value Vg from continual value Wg by one continuation term.
Hierarchical RL has been speaking two dialects that barely meet. One treats the goal as part of the task: the environment or the designer names G, and the agent only picks actions. The 2026 ACGVF construction of Nangue Tasse et al. hands the agent a goal Gt and a done bit Dt as well, chosen in service of environmental reward. It assumes full observability and writes the policy on world state s. Murphy's 2025 generic agent goes the other way: the policy π(a|z,g) reads an internal memory z that is updated from observations, but g is still an external input.
Each piece is missing what the other has. ACGVF has agency over goals and stopping, and no information state. The belief-state agent has memory, and cannot change its own goal. This note wires them together with the hierarchical hidden Markov model (HHMM) of Murphy and Paskin (2001). There are no new experiments. The contribution is a call-return grammar written down carefully.
Internal state now has two timescales. The goal Gt is piecewise constant. The within-goal state Zt updates after every observation. The done bit Dt is the boundary: if Dt=0 the goal is copied; if Dt=1 an upper policy μη picks a new goal from the latest Z. That is stronger than resampling a goal label every step. The goal persists, and behaves like an option in the sense of Sutton et al. (1999).
The joint factors in this order: (Zt, Gt) emit (At, Dt), the environment emits the next observation and reward, Z updates, then G updates. Three learnable pieces sit on the agent side: a goal-conditioned policy πθ(a,d|z,g), a state update qϕ (a deterministic recurrent map or a filter), and a goal selector μη that fires only at boundaries.
One change relative to the classical HHMM of Fine et al. (1998): entering a new high-level state used to redraw the child from a top-down prior. An agent should not wipe its world belief when it changes goals, so Zt+1 always depends on Zt. The next goal is chosen after Z is updated, which flips the vertical arrow in the usual DBN drawing. The role is flipped too. This HHMM is an observation-conditioned generator of actions, not an action-conditioned generator of observations.
Under partial observability, goal satisfaction cannot be a predicate of the hidden transition (s,a,s'). It is replaced by an internal satisfaction model σψ in [0,1]. Continuing or terminating correctly keeps environmental reward; terminating incorrectly gets a lower-bound penalty rmin. The within-goal Qg and Vg recover the ACGVF Bellman equation with s replaced by z. The lower controller learns mastery: reach the goal and notice that it has been reached. The upper controller then chooses among mastered goals.
Segment value Vg puts zero continuation after d=1 and scores one goal-directed stretch. Continual value Wg adds an upper-level continuation H(z)=maxg Wg(z) at termination. Two actions that finish the current goal with the same immediate reward can leave the agent near many valuable future goals or trap it. Vg treats them as equal; Wg prefers the first. The macro-transition at a boundary is semi-Markov, with duration Δk entering as γ^{Δk}. The same object is an HHMM in primitive time and an option-like semi-MDP at goal boundaries. The option label, the termination decision, and the latent information state are all internal.
Learning has two readings. If Gt and Dt are unrecorded, as in goal discovery from demonstrations, equation (3) is a controlled HHMM likelihood and discrete states admit an E step linear in sequence length. If the agent chooses G and D, those variables are observed from its point of view: TD learns Qg or Wg, actor-critic trains both levels, and μη is updated on duration-aware boundary returns. A duration regularizer Ωdur blocks the two degeneracies of always-done and never-done.
A convenience result: for a finite goal set the upper critic is the μη-weighted average of the lower goal-conditioned values, so one parameter block can be dropped, the same relation option-critic uses. Boundary states are not distributed like arbitrary states, so a shared critic may be least accurate exactly where the upper advantage is evaluated.
There are no experimental numbers and no baselines. The claims that can be checked are structural:
| Object | Role |
| Gt | piecewise-constant goal, copied when Dt=0 |
| Zt | per-step information state, kept across goals |
| Dt | return from the within-goal controller |
| Vg | segment value with zero future after done |
| Wg | continual value with continuation H(Zt+1) |
The joint loss adds a satisfaction term, a TD term, on-policy critics, environmental return J, and the duration regularizer, with λ weights. J grounds goal selection in environmental reward. TD teaches mastery with the internal reward that penalizes false termination. Actor and critic have to share a reward function, or the advantage is meaningless.
Anyone working on goal-conditioned RL, options, or hierarchical POMDPs gets a wiring diagram. Persistent intentions under partial observability do not need a new notation. HHMM call-return is already that grammar. Credit assignment splits cleanly: primitive rewards train the lower controller, boundary-to-boundary returns train the goal selector. Deeper stacks add HHMM levels with the nesting constraint that a parent may move only after every descendant has returned.
This is a formal unification, not a new algorithm. Discrete versus continuous Z, whether to use a duration prior, and where satisfaction labels come from are still open implementation choices.
The note barely discusses empirical limits, so the gaps are the limits. There is no simulation. Whether σψ, the duration regularizer, and the two reward functions can coexist in a deep net without gradient fights is unknown. When no independent satisfaction target exists, the text says to use weak supervision or a latent-variable objective and stops there. The upper critic can be dropped, and the training-distribution caveat is one sentence. The claim that ACGVF subsumes almost all prior RL, control, and planning is conceptual; this note does not prove it. Treating (Z,G) as a sufficient statistic is exact for a true belief state and only an approximation for a learned Z, with no measurement of the gap.