Oxford/UCL: simple-to-complex curricula work by unlearning, in humans and networks alike

Curriculum unlearning in humans and neural networks

Jirko Rubruck, Alice Zhang, Andrew Saxe, Christopher Summerfield

cs.LG, q-bio.NC

2026-11-03

Oxford/UCL: simple-to-complex curricula work by 'unlearning', uncompressing rule representations that early training collapsed, in both humans and neural networks.

What problem this solves

Teaching simple rules before exceptions is standard practice in education, and experimental psychology has repeatedly confirmed that it speeds human learning. What has been missing is a computational account of why. Deep learning makes the gap more glaring: curriculum benefits in supervised learning are inconsistent and paradigm-dependent, so the default remains i.i.d. random sampling. A team from Oxford's Department of Experimental Psychology and UCL's Gatsby Unit/SWC proposes a mechanism: curricula do not mainly teach new content, they help learners unlearn simple rules that earlier learning compressed into representation. The authors call this curriculum unlearning.

Method

The task is the classic rule plus exception (R+E) categorisation problem: bug stimuli vary on three binary features, six of the eight stimuli can be labelled from a single feature, and two exceptions require combining the other features. Three conditions differ only in ordering, not total exposure: no curriculum (roughly 75/25 rule/exception per block), simple-to-complex (exception fraction rising from 0 to 0.5 across five blocks), and complex-to-simple (the reverse), 240 trials each. In Exp. 1, 207 humans learned trial by trial with feedback, then were tested without it.

Each participant was paired with a two-layer network (32 ReLU hidden units, small init σ=0.01, single-trial gradient descent) in a yoked design: the network saw exactly the same 240-trial sequence as that participant. Gaussian noise was injected into hidden activations, and a sweep over learning rate and noise picked the config that best reproduced human learning curves (lr 0.30, noise 1.0, RMSE 0.077).

The theoretical starting point is simplicity bias: gradient-based learning from small weights captures the low-dimensional directions that explain the most variance first, and discarded dimensions produce near-zero activations and weak gradients, making them hard to recover later. Biological learners do not use gradient descent but show the same stubbornness: participants who first learn that colour is irrelevant can go hundreds of trials without noticing that colour fully predicts the answer. The proposed role of a curriculum is to pry that compressed dimension back open.

Validation runs on three levels. Behaviourally, ten candidate policies are fit per block to humans and networks by BIC. For representations, the participation ratio of the eight hidden-layer stimulus representations measures effective dimensionality, and representations are projected onto a rule axis and an exception axis; in humans, the reaction-time cost of exception trials serves as a proxy. Causally, Exp. 2 (n=1027) manipulates the prior through pretraining (a 1-D rule compresses network representations to PR 1.09, XOR keeps them at PR 2.93) and then asks who gains more from a curriculum.

Results

ComparisonMetricResult
Humans, rule+exception strategy at testsimple-to-complex vs none65.3% vs 44.4% (+20.8 pp, p=.024)
Humans, rule → rule+exception transitionsimple-to-complex vs none39.0% vs 30.4% (pooled OR=2.23 vs other conditions, p=.014)
Networks, exception accuracy at testsimple-to-complex vs none+7.1 pp (p<.001)
Networks, final participation ratiosimple-to-complex vs none2.37 vs 2.27 (+0.104, p<.001)
Networks, curriculum benefit on exceptions1-D pretraining vs XOR pretraining+0.051 vs +0.018 (diff 0.033, p=.004)

Both humans and networks fall into the simple rule first. Under no curriculum, human exception accuracy dips below chance early (first-block mean 0.44), and 63 of 63 no-curriculum networks spent the first half of training below chance on exceptions, the signature of classifying every stimulus by the rule. Simple-to-complex repairs this; complex-to-simple actively hurts (human exception log-odds −1.52, p<.001) and leaves only 19.4% of people on the optimal strategy at test. Network strategy distributions correspond to the humans they were yoked to: distance 0.091 to same-curriculum participants, below all 5,000 permutations of a reassignment null (p<.001).

The mechanism is selective. The curriculum adds +1.58 of separation along the exception axis (p<.001) with no difference on the rule axis (+0.19, p=.28), and final PR correlates with final exception accuracy at r=0.70. Two controls pin down causality. After appending an identical balanced block to all conditions, the exception benefit survives (+0.040) while the rule cost and the complex-to-simple harm vanish, ruling out recency. With the hidden layer frozen and only the readout trained, both pretraining groups benefit equally (+0.111 vs +0.113, p=.90): the effect requires a compressed representation, not merely carried-over readout weights.

The human causal test mirrors the networks. Participants pretrained on the 1-D rule gained significantly more from the curriculum on exceptions than XOR-pretrained participants (three-way interaction log-odds 0.10, p=.020); the share reaching rule+exceptions rose from 41.7% to 78.8% after 1-D pretraining versus 51.4% to 73.4% after XOR. Participants who compressed more during pretraining (an RT-based selective-attention measure) benefited more (log-odds 0.20, p=.015), while compression alone did not predict overall test accuracy (p=.64). The exception benefit was positive in 34 of 35 hyperparameter configurations.

Why it matters

This turns "curricula work" from an empirical regularity into a testable mechanism: the value of a curriculum depends on how much there is to unlearn. The transferable hint for practitioners is that curriculum gains should scale with how compressed the initial representation is. A learner biased toward low-dimensional, underfit solutions has more to gain from careful ordering; one that already represents the task in high dimensions may gain nothing. It also offers a candidate explanation for hit-or-miss curriculum results in the deep learning literature: initial compression was never controlled.

Honest framing: this is a cognitive-science result on a three-feature, eight-stimulus task, with effect sizes of a few percentage points. It is not a training recipe for large-scale pipelines. The value is the mechanism and its prediction, plus a clean demonstration that a two-layer network with hidden noise reproduces human trial-by-trial learning curves closely (RMSE 0.077).

Limitations

The authors list their own: the network lacks the hippocampus/neocortex division, and the hippocampus is thought to be exactly the circuit for learning exceptions; networks show forgetting on rule stimuli that humans do not, and their curriculum benefits are smaller overall; when exceptions are sufficiently rare, the rule cost can offset the curriculum advantage entirely. Points that remain open: generalisation beyond three binary features is untested; human "dimensionality" rests on RT proxies rather than neural measurement; the yoked fit leans on two degrees of freedom (hidden noise and lapse rate), so "networks mimic humans" and "humans use the same mechanism" remain one inference apart.

Terms

Source

What people are saying

Related papers

All paper explainers