Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection
Mosh Levy, Aly Lidayan, Yoav Goldberg
cs.LG, cs.AI
2026-10-20
JEPA picks the feature set with the most predictive information, and reconstructing only C (β=0.36) lifts unsupervised D from 61.3% to 85.5% at copy probability 0.60.
A JEPA (joint-embedding predictive architecture) maps each observation to a latent vector and trains a predictor to forecast the next latent. The loss does not reconstruct pixels, and it does not name which features to keep. A small latent has to drop something. Dropped features are invisible to later planning.
Per-feature accounts keep what changes slowly or predicts itself, and drop what is pure noise one step ahead. A traffic light would lose that contest, because its next color cannot be predicted. The current color still decides whether a cyclist rides or stops. Drop the light and the error lands on the cyclist.
Predictive closure scores sets of features. Regularization pulls the latent distribution toward a chosen target and pins an entropy budget. A set is feasible when an encoder can read every feature in it and the set's entropy matches that budget. The encoder that drives the loss as low as it will go selects a feasible set with maximal predictive information, the mutual information between the set's values now and at the next step.
With the budget fixed, prediction loss falls when the next latent is more determined by the current one. That raises mutual information between consecutive latents. The latent and the features readable from it determine each other, so that quantity is the set's predictive information.
The same criterion minimizes two terms. Aleatoric uncertainty is the entropy still left in the next values once the whole current observation is known. Predictive leakage is the extra information the full observation provides about those next values, beyond the set's own present. A current value that helps forecast its partners cuts leakage and can pay for the noise in its own future.
When features evolve on their own, each feature's contribution and capacity cost ignore its companions, so ranking by self-predictability is enough. Once a current value forecasts someone else's next step, a light whose next color is pure noise can still sit in the winning set, if its current color cuts leakage on the images.
A mutually predictive group that exceeds capacity becomes less readable, since every subset that fits still leaks through the features left out. An auxiliary loss or predictor side information aimed at one feature also strengthens its predictive partners.
The sequences are CIFAR-10 collages, the latent has three dimensions, and the loss is 0.9 times the prediction loss plus 0.1 times SIGReg. Probes read only the frozen main encoder, on another 5,000 class-balanced images held out from encoder training. Chance is 50%. Reconstruction uses one seed; the other settings use three.
Two images and a binary light: exactly one image is replaced each step, and each image persists with probability 0.5 at every q. The next light is uniform, so storing it always costs one bit of aleatoric uncertainty, while with probability q the current light picks which image is replaced. As q moves from 0 to 1, the light is decoded more accurately. The interaction is nonlinear. The paper does not list pointwise accuracies for this curve.
In a four-tile collage, pairs A/B and C/D copy only inside the pair. Raising only pD, the probability that C is copied onto the next D, from 0.60 to 1, shifts decoding from A/B toward C/D. Rules inside A/B stay fixed, and the two tiles in a pair decode similarly. The shift holds at regularization weights r of 0.05, 0.10, and 0.20. At r = 0.20, mean C/D accuracy can fall from pD = 0.95 to pD = 1.
A/B keeps a bidirectional copy probability of 0.85. A competing cycle grows through N = 2, 4, 8, 16, with average copy probability 0.93 and pixel area changing by less than 1.2%. Mean detection on the cycle drops at every step up in N, in all three seeds. At N = 16, A/B is more readable in all three seeds, at both the worst and the best positions. At r = 0.05 that reversal appears in one run of three.
Reconstructing only a coarse 4×4 view of the current C image, at r = 0.10:
| Image | pD=0.60, β=0 | pD=0.60, β=0.36 | pD=1, β=0 | pD=1, β=0.36 |
| A | 86.1% | 63.1% | 53.0% | 51.0% |
| B | 85.5% | 62.6% | 52.8% | 50.6% |
| C (reconstruction target) | 62.0% | 87.0% | 86.8% | 89.0% |
| D (never reconstructed) | 61.3% | 85.5% | 86.7% | 88.5% |
At β = 0.36, D stays close to C for all five values of pD. The high-pD baseline already favors C/D. At r = 0.20 and pD = 0.75, moving β from 0.135 to 0.36 lowers C and D together.
Feeding C through a second encoder of the same architecture, into the predictor, leaves the main latent three-dimensional. Across three seeds, D rises and B falls for most pD, and A rises across the sweep. D is more readable at pD of 0.75 and above, and less readable at 0.60.
The predictor separates the true next observation from an alternative the transition rules forbid. Chance on that choice is 50%. The light at q = 1 scores 77.13% ± 0.51, against 49.97% ± 0.43 after shuffling predictions. Two-pair models without side information score 88.88% ± 0.35 at pD = 0.60 and 96.99% ± 0.49 at pD = 1. Supplying C moves those to 91.27% ± 0.19 and 97.96% ± 0.13.
At pD = 0.60, supervising only C moves D from 61.3% to 85.5%. If the feature you want has no usable loss, supervise a partner that forecasts it, or pass that partner to the predictor. The paper's example is velocity: it fixes where an object will be, so a loss on velocity can make position more readable. Features you did not name still move. In the side-information run, A improves and B gets worse.
Treating actions as side information, an action-conditioned JEPA would favor features the agent can change. No such model is trained here.
The same accounting matches earlier observations. A small pattern held fixed inside a sequence can fill the latent. A cue resampled every step, which does not help predict other features, drops out even when it decides which action is rewarded. V-JEPA 2 is not retrained here, and there is no planning score.
The collages test the direction of each effect. Magnitude on natural video or in a large latent is unmeasured. The three-dimensional latent is small on purpose, so these simple sequences still run out of room.
Reconstruction is a single seed, and a larger β can make C and D worse together. Side information splits A from B, while the set objective prices them as a pair. A low readout from the main latent does not mean the predictor lacks the feature. The side encoder can still carry it.
Negative probe examples remove the target image from every position, so an unordered pair in the latent can pass both position detectors. As N grows, tiles shrink from 90 pixels to 32, and capacity is tied to how hard the image is to extract.
Prediction loss equals mutual information under isotropic Gaussian errors. Without that assumption the loss maximizes a lower bound. The derivation also assumes every encoder matches one shared Gaussian exactly. The rereading of Sobal, Enigma, Karmann, and Pendharkar is not a rerun of their setups.