Teaching a local Qwen 3.5 4B a new domain: CPT vs RAG, a four-phase experiment with evals

funJS · reddit · 2026-09-25

The author ran a series of experiments teaching a local Qwen 3.5 4B a new domain (Unsloth for both CPT and SFT LoRA), in four phases:
Phase 1: continued pretraining (CPT) for domain injection, focusing on picking training sets that generalize to unseen questions.
Phase 2: comparing reasoning performance of internalized knowledge (CPT) vs RAG-injected content.
Phase 3: practical combination — enriching CPT-trained knowledge with RAG instead of treating them as competing approaches.
Phase 4: a comprehensive eval strategy, including SFT-tuning the CPT model to output strict schemas instead of English sentences to simplify eval checks.
Full write-up on the author's blog; directly useful for anyone doing local domain fine-tuning.

More from coding & agent