Pairwise BCE lifts continual-learning accuracy from 19.4% to 58.1%, and Adam forgets more than SGD
BlackHC · x · 2026-09-16
The author ran an automated research experiment (done with Claude) on a toy continual-learning setup: a small MLP learns Split MNIST sequentially (0/1 → 2/3 → … → 8/9), each stage using only that pair's training images, then gets tested on all ten digits (averaged over 3 seeds).
- Cross-entropy over all ten outputs + SGD ends at 19.4% accuracy — yet restricting the same model to choose within the correct pair yields 97.5%, showing poor ten-class accuracy can hide retained within-pair discrimination
- Switching to binary cross-entropy on only the current pair's two outputs raises final ten-class accuracy to 58.1%; old output weights get no direct updates, but predictions can still change as the shared latents evolve
- Optimizer comparison: with Adam the numbers are 19.6% vs 71.3% — Adam causes more "forgetting" than SGD
Related event: BlackHC's toy continual learning experiment: PCA replay hits 91.1%(6 posts)→
More from Research
- Teaching a robotic hand to walk on its fingertips, no robot arm required — Scobleizer · 2026-09-16
- After searching thousands of learning rules, none beat backprop—and Nature confirms it's biologically plausible — aran_nayebi · 2026-09-16
- BoltzMol-1 finds WRN inhibitor hits for ~$10K in 10 days; best IC50 8.4 µM — GabriCorso · 2026-09-16
- Researchers clash over Sakana AI's bioplausible learning claim: MNIST results don't count — aran_nayebi · 2026-09-16
- A 0.62-AUC model made a 65,578-candidate materials search tractable, doubling hit rate — bravo_abad · 2026-09-16
- NeurIPS reference checker flags LLM-corrupted bib entries; desk rejection feared — suryanreddy · 2026-09-16