GPT-2 'sleep' replay from a frozen snapshot keeps 58% of old facts vs 6% without
Junior_Reputation639 · reddit · 2026-11-21
A small-scale experiment tested sleep-inspired generative replay for catastrophic forgetting in GPT-2 (124M), full fine-tuning: train on 120 made-up facts to 98% recall, then 120 new facts with 25% of each batch used for replay, 3 seeds.
Results: no replay retained only 6% of old facts; dreaming once before learning gave 40%; replaying dreams from a frozen saved copy kept 58%; repeatedly dreaming from the model being trained only 9%, because its dreams degraded into false facts.
Key takeaway: the source of the 'dreams' matters — use a frozen snapshot. Full write-up at evo-quest.github.io; code available. Next steps: Qwen2.5-0.5B with LoRA and replaying general text alongside facts.
More from Research
- Hassabis predicted 2 years for AI to crack a Millennium Prize problem; 6 months later, both Riemann and Navier–Stokes have proposed solutions — Hesamation · 2026-11-21
- ERC reviewer rejects LLM autoformalization proposal: "tried without good results" — marc_lelarge · 2026-11-21
- Terence Tao on AI math: 'strip mining' that wins proofs but teaches us nothing — yacineMTB · 2026-11-21
- Anthropic claims Claude proved the Riemann hypothesis; Stanford posts all-human rival proof hours later — The Decoder · 2026-11-21
- Claude's Riemann Hypothesis proof called 'almost certainly true' by mathematicians — Just-Grocery-2229 · 2026-11-21
- 41000 measurements show scientific errors are heavy-tailed, 5σ outliers vastly exceed Gaussian odds — tetsuoai · 2026-11-21