Paper: concept circuits track how LLMs learn and forget during continual pre-training
tokenbender · x · 2026-08-24
A new arXiv paper, "How Do Large Language Models Learn Concepts During Continual Pre-Training?" (UC Davis, Virginia Tech, UCLA, Meta AI), studies how LLMs acquire, retain, and forget concepts during continual pretraining.
Key findings:
- Concept circuits — internal computational subgraphs tied to specific concepts — provide a non-trivial, consistent signal of concept learning and forgetting;
- Concept circuits show a stage-wise temporal pattern: early increase, gradual decrease, then stabilization;
- Concepts with larger learning gains tend to exhibit greater forgetting under subsequent training;
- Semantically similar concepts induce stronger interference than weakly related ones;
- Conceptual knowledge differs in transferability, with some concepts significantly facilitating the learning of others.
More from Research
- NVIDIA's TacitVLA: robots reason without reading their own thoughts, 3.4x lower latency — arankomatsuzaki · 2026-10-16
- Cohere Labs world model course session 4: DreamerV1-V4 architectures in one go — Cohere_Labs · 2026-10-16
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
- One tiny word changes the conversation: how LLMs pick up on 'just' — StellaLisy · 2026-10-16
- SEAR: first large-scale controlled study of self-evolving robot agents, 7 LLMs, 330 tasks — xiye_nlp · 2026-10-16
- After Claude-assisted elliptic curve records fell, EC Rank challenge opens 156 record slots — Jsevillamol · 2026-10-16