NYU's Epiplexity Metric Beats SOTA Curriculum Selectors on OOD Generalization

Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization

Ellen Su, Andres Potapczynski, Shikai Qiu, Edward Hughes, Andrew Gordon Wilson

cs.LG, cs.CL

2026-08-12

NYU turns epiplexity into a training signal for data selection and synthetic data generation, consistently beating SOTA curriculum baselines in both settings.

What problem this solves

Data-selection methods like DoReMi and ADO show empirically which training-data mixtures improve downstream performance, but none address a deeper question: what property makes data generalize well to tasks unseen during training? The authors start from epiplexity, a recently proposed measure of how much structural information a compute-bounded learner can extract from data. Data rich in reusable circuits and subprograms, the hypothesis goes, should transfer better to new domains than either trivial or unlearnable data. This paper turns that hypothesis into an online training signal.

Method

The practical estimate of epiplexity is simple: the area between a training loss curve and its final floor value. Data that's too easy (repetitive sequences) drives loss to near zero almost immediately, leaving no area. Data that's unlearnable (pure noise) never reduces loss at all. Only data that causes sustained, substantial loss decreases has high epiplexity.

Two applications follow:

EpiSelect (data selection). The team fits a cross-domain scaling law predicting each domain's loss as a function of tokens seen from every domain, then uses that law to estimate the marginal epiplexity gain from training on more tokens of domain j. A softmax over that gain becomes the sampling weight for the next batch, continuously reweighting the curriculum toward domains still yielding epiplexity gains.
EpiGen (synthetic data generation). A generator is trained via REINFORCE, with the reward defined as the change in a learner's epiplexity, measured on a buffer of previously generated data, before and after training on the current batch. The buffer isn't used for training the learner itself; it's a reference distribution for computing the reward.

Results

Setting	Comparison	Result
Pile data selection (124M/1.3B, 10 zero-shot tasks)	EpiSelect vs. Natural (no selection)	0.394 vs 0.377 / 0.431 vs 0.422
Pile data selection	EpiSelect vs. ADO (prior SOTA)	0.394 vs 0.379 / 0.431 vs 0.425
GLUE fine-tuning after GPT-2 synthetic pretraining	EpiGen vs. Pretrained baseline	0.770 vs 0.743 (+2.7 pts)
Mixing synthetic + real data (OOD GLUE)	50% synthetic + 50% OWT	0.764, beating both pure synthetic (0.752) and pure real (0.757)

EpiSelect tops 6 of 10 downstream tasks at both model scales. A side finding backs up the intuition: the fitted cross-domain influence matrix is diagonally dominant, meaning tokens from one domain mostly reduce loss within that same domain rather than transferring to others, so most of the structural information epiplexity picks up doesn't cross domain boundaries easily.

One benchmark-level gotcha surfaced along the way: on the classic Pile dataset, PileCC (general web crawl) makes up 41% of all tokens, and training on PileCC alone beats every curriculum method the authors tested, including the prior SOTA. That makes Pile useless for distinguishing selection methods, so the team reran everything on the more balanced Common Pile (8TB, 30 sources).

For synthetic data, the epiplexity-guided improvement only shows up when both generator and learner start from pretrained weights. Swap in random initialization for both, and average GLUE score barely moves (0.743 to 0.740), indicating the method amplifies capability already present in the pretrained model rather than manufacturing a signal from nothing.

Why it matters

For teams designing pretraining data mixtures, this offers an online selection signal that needs no separate proxy model and no hand-tuned curriculum rules, computed directly during the real training run with no offline search overhead. For synthetic-data teams, this is a rare scoring mechanism for purely generated data that doesn't depend on an external verifier like a code executor or math checker, a gap that's mostly gone unaddressed. That said, everything here is validated at 124M-1.3B parameters; whether it holds at mainstream training scale (billions to tens of billions) is untested.

Limitations

The authors flag three gaps themselves: all experiments are on small models, so scaling behavior is unverified; both applications rely on a tractable proxy (the prequential coding estimate) rather than the formal epiplexity quantity, with no theoretical bound on how far the proxy can drift from the real thing; and the current setup only covers single-modality, single-objective generation, leaving multi-modal and multi-objective synthesis to future work. The core causal claim also rests mainly on a correlation (Pearson r=0.88 between domain-level epiplexity and OOD accuracy), not a controlled intervention, though the authors do rule out one obvious confound by showing model weight norm has no comparable correlation (r=0.01, not significant).

Terms

Epiplexity: a measure of how much structural information a compute-bounded learner can extract from a dataset, roughly the area under the training loss curve above its final value.
REINFORCE: a policy-gradient RL algorithm that adjusts a generative model's output probabilities directly from a scalar reward signal.
Scaling law: an empirical formula describing how model loss changes with data volume, parameter count, etc.; here used to forecast how much further loss will drop per domain.
OOD (out-of-distribution) generalization: how well a model performs on tasks or distributions it wasn't trained on.