DEFT proposes a structural way to prevent catastrophic forgetting in continual learning
TheGradient · x · 2026-07-24
What the thread says
DEFT is presented as a continual/transfer learning method designed to avoid catastrophic forgetting. The core idea is to identify redundant neurons in the frozen “Core,” merge them to recover capacity, and then isolate the trainable “Slack” with an architectural mask so updates cannot flow back into the Core.
Why it matters
The author frames this as a cleaner alternative to older anti-forgetting setups that rely on stop-gradient tricks, EMA teacher-student models, frozen pretrained encoders, or VICReg-style objectives with many hand-tuned coefficients. The pitch is that DEFT uses a single structural mechanism to protect prior knowledge while still allowing new learning.
Related event: DEFT Introduces Core-Slack Architecture to Prevent Catastrophic Forgetting(3 posts)→
More from Research
- AI slop is already clogging PR review and weakening the credit system behind science — rbhar90 · 2026-07-27
- ICML 2026 oral paper replication scores stay middling after a stricter re-scoring — profjamesevans · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Seed IQ navigates Doom II, prompting questions about benchmarks beyond ARC-AGI — Fit_Transition8824 · 2026-07-27
- Agentic Data Science in Practice: Agents Write Code but Answer Wrong Questions — hugobowne · 2026-07-27
- A concise canon of foundational papers in ML, systems, NLP, speech, and audio — deliprao · 2026-07-27