Teaching a local Qwen 3.5 4B a new domain: CPT vs RAG, a four-phase experiment with evals
funJS · reddit · 2026-09-25
- The author ran a series of experiments teaching a local Qwen 3.5 4B a new domain (Unsloth for both CPT and SFT LoRA), in four phases:
- Phase 1: continued pretraining (CPT) for domain injection, focusing on picking training sets that generalize to unseen questions.
- Phase 2: comparing reasoning performance of internalized knowledge (CPT) vs RAG-injected content.
- Phase 3: practical combination — enriching CPT-trained knowledge with RAG instead of treating them as competing approaches.
- Phase 4: a comprehensive eval strategy, including SFT-tuning the CPT model to output strict schemas instead of English sentences to simplify eval checks.
- Full write-up on the author's blog; directly useful for anyone doing local domain fine-tuning.
Related event: Hands-on: Teaching a Local 4B Model New Domain Knowledge via CPT+RAG(5 posts)→
More from coding & agent
- Too many 'X times faster' claims hide extremely narrow benchmarks, says Ruff author — charliermarsh · 2026-09-25
- Ruff author Charlie Marsh: microbenchmark culture is getting worse — charliermarsh · 2026-09-25
- Claude API adds inline tool definitions for adding full tools mid-conversation — EricBuess · 2026-09-25
- Hands-on repo teaches GitHub stacked PRs with gh stack CLI and an AI coding agent workshop — DanWahlin · 2026-09-25
- Energy's AI agent can log in and pay for you, with credentials kept local and hidden from LLMs — emilahlback · 2026-09-25
- OpenRouter launches unified web search server tool with per-provider pricing — jeff_weinstein · 2026-09-25