Teaching a local LLM a new domain with CPT + RAG: a four-phase experiment with evals
funJS · reddit · 2026-09-25
The author shares a series of experiments teaching a local LLM (Qwen 3.5 4B, trained with Unsloth via CPT and SFT LoRA) a new domain, in four phases:
- Phase 1: Using continued pretraining (CPT) to internalize domain knowledge, focusing on picking training sets that generalize to unseen questions.
- Phase 2: Comparing reasoning performance between internalized knowledge (CPT) and RAG-injected content.
- Phase 3: The practical takeaway — combine CPT-trained knowledge with RAG instead of treating them as competing approaches.
- Phase 4: A comprehensive eval strategy, including SFT-tuning the CPT model to output strict schemas instead of English sentences, simplifying automated checks.
Full write-up is linked on the author's blog.
Related event: Hands-on: Teaching a Local 4B Model New Domain Knowledge via CPT+RAG(5 posts)→
More from Research
- humans& launches Persimmon, a 550B-parameter user model that simulates real human behavior — CShorten30 · 2026-09-25
- Protein design must account for neighbors: 578 variants show similarity predicts function poorly — bravo_abad · 2026-09-25
- PhAILabs launches ScienceBuddy, a free AI-scientist workspace offering GPT-6 at no cost — udmrzn · 2026-09-25
- Nautilus accepted to NeurIPS: one prompt, any policy, any benchmark, any robot — GeorgiaChal · 2026-09-25
- SkillRL (NeurIPS 2026): 7B model beats GPT-4o by 41% via recursive skill evolution — cihangxie · 2026-09-25
- NVIDIA's open-source Nemotron-Cascade RL recipe wins NeurIPS Oral, IOI silver — _weiping · 2026-09-25