Hands-on: Teaching a Local 4B Model New Domain Knowledge via CPT+RAG
Author @funJS shares a full hands-on retrospective on using continued pre-training (CPT) to teach a local small model entirely new domain knowledge. The experiment is based on the Qwen 3.5 4B model, uses Unsloth for both CPT and SFT LoRA training, and pairs it with RAG for comparison, with evaluation results provided throughout.
Confirmed
- Model and tools: Qwen 3.5 4B, with Unsloth for CPT and SFT LoRA
- The experiment proceeds in four stages:
- Stage 1: teaching the model new domain knowledge via CPT, with a focus on how to pick training sets that generalize to unseen questions
- Stage 2: comparing "internalized knowledge" (CPT) against RAG
- Stages 3 and 4: exploring hybrid approaches combining CPT and RAG, plus an overall evaluation (the post summary is truncated; see the original post for details)
Why it matters
- The series offers an end-to-end practical path from training-set construction through continued pre-training to evaluation, directly useful for developers looking to inject domain knowledge into local small models
- The CPT-vs-RAG comparison and hybrid experiments address the common engineering trade-off of whether knowledge should be internalized into parameters or handled via external retrieval
2026-09-25 ~ 2026-09-25 · 5 related posts
Primary sources
- Teaching a local LLM a new domain with CPT + RAG: a four-phase experiment with evals — funJS · 2026-09-25
- [source] Teaching a Local 4B LLM a New Domain: CPT vs RAG Experiments With Evals — funJS · 2026-09-25