GPT-2 'sleep' replay from a frozen snapshot keeps 58% of old facts vs 6% without

Junior_Reputation639 · reddit · 2026-11-21

A small-scale experiment tested sleep-inspired generative replay for catastrophic forgetting in GPT-2 (124M), full fine-tuning: train on 120 made-up facts to 98% recall, then 120 new facts with 25% of each batch used for replay, 3 seeds.

Results: no replay retained only 6% of old facts; dreaming once before learning gave 40%; replaying dreams from a frozen saved copy kept 58%; repeatedly dreaming from the model being trained only 9%, because its dreams degraded into false facts.

Key takeaway: the source of the 'dreams' matters — use a frozen snapshot. Full write-up at evo-quest.github.io; code available. Next steps: Qwen2.5-0.5B with LoRA and replaying general text alongside facts.

Original post →

More from Research

Research channel →