Paper: concept circuits track how LLMs learn and forget during continual pre-training
tokenbender · x · 2026-08-24
A new arXiv paper, "How Do Large Language Models Learn Concepts During Continual Pre-Training?" (UC Davis, Virginia Tech, UCLA, Meta AI), studies how LLMs acquire, retain, and forget concepts during continual pretraining.
Key findings:
- Concept circuits — internal computational subgraphs tied to specific concepts — provide a non-trivial, consistent signal of concept learning and forgetting;
- Concept circuits show a stage-wise temporal pattern: early increase, gradual decrease, then stabilization;
- Concepts with larger learning gains tend to exhibit greater forgetting under subsequent training;
- Semantically similar concepts induce stronger interference than weakly related ones;
- Conceptual knowledge differs in transferability, with some concepts significantly facilitating the learning of others.
More from Research
- NVIDIA Releases ARDY: Real-Time Interactive Human Motion Generation Model — rsasaki0109 · 2026-08-27
- Menlo Achieves Zero-Shot Sim2real for Asimov Biped, Citing Unitree Early-2025 Parity — chris_j_paxton · 2026-08-27
- OpenWiki 0.4.2 Flaw Found: Adversarial Proof Reveals History Leak — tallmetommy · 2026-08-27
- Meitu MT Lab Presents CFT for Stable Portrait Relighting at ECCV 2026 — jiqizhixin · 2026-08-27
- Visual General Intelligence White Paper: Vision as a Pathway to AGI — zhenjun_zhao · 2026-08-27
- Expert Warns Against Self-Deception in ML Evaluation — PolarBearby · 2026-08-27