BlackHC's toy continual learning experiment: PCA replay hits 91.1%

@BlackHC released a full ladder of continual learning toy experiments, complete with code, plots, and research logs. He had Claude autonomously research a toy problem: a small MLP learns Split MNIST in the order 0/1 → 2/3 → … → 8/9, training only on the current digit pair at each stage, then testing on all ten classes (averaged over 3 seeds) to see whether the network can learn new tasks without forgetting old ones.

Confirmed

Switching to a per-pair BCE loss raised ten-class accuracy from 19.4% to 58.1%: although old output weights receive no direct updates, the shared hidden layers keep changing, so predictions drift — hence the need to penalize changes to old predictions.
A subtle tuning pitfall was found: changing BCE from mean-reduction to sum-reduction actually dropped accuracy by 7 points.
Generative PCA replay: each class is compressed into a mean vector plus 10 PCA directions over pixels, sampled to synthesize inputs; combined with storing old predictions and penalizing drift, this reaches 90.2%; with SGD + PCA replay, raising per-task training from 2 to 4 epochs reaches 91.1%.
Reference point: standard joint training with 10 passes over all task data achieves 95.2%, though the compute budgets differ.

Why it matters

The experiments demonstrate a complete LLM-driven autonomous research loop on a small problem (task design → loss fix → replay mechanism → tuning), and highlight that loss implementation details (sum vs. mean) can silently cost several points — a caution for reproducing others' continual learning results.
The author explicitly notes this is only a toy benchmark with just 3 seeds, and the results should not be over-generalized.

2026-09-16 ~ 2026-09-16 · 6 related posts

Primary sources

Continual learning log: SGD + PCA replay hits 91.1%, full code released — BlackHC · 2026-09-16 20:57 (UTC+08:00)
Same toy benchmark, different budgets: PCA replay 91.1% vs joint 95.2% — BlackHC · 2026-09-16 20:57 (UTC+08:00)
BCE averaging convention was secretly tuning your learning rate — BlackHC · 2026-09-16 20:57 (UTC+08:00)

A Small MLP Learns Digits 0-9 in Order: Toy Continual Learning Results — BlackHC · 2026-09-16
Pairwise BCE lifts continual-learning accuracy from 19.4% to 58.1%, and Adam forgets more than SGD — BlackHC · 2026-09-16
[source] BCE averaging convention was secretly tuning your learning rate — BlackHC · 2026-09-16
Generative PCA replay: mean + 10 PCA directions per class hits 90.2% — BlackHC · 2026-09-16
[source] Same toy benchmark, different budgets: PCA replay 91.1% vs joint 95.2% — BlackHC · 2026-09-16
[source] Continual learning log: SGD + PCA replay hits 91.1%, full code released — BlackHC · 2026-09-16