Teaching a Local 4B LLM a New Domain: CPT vs RAG Experiments With Evals
funJS · reddit · 2026-09-25
- Author documents a four-phase project teaching a local Qwen 3.5 4B model a new domain via continued pretraining (CPT), using Unsloth for both CPT and SFT LoRA training
- Phase 1: picking a training set that generalizes to unseen questions
- Phase 2: comparing reasoning via internalized CPT knowledge vs RAG-injected content
- Phase 3: a practical hybrid combining CPT-trained knowledge with RAG instead of treating them as rivals
- Phase 4: a full eval strategy, including SFT fine-tuning the model to output strict schemas to simplify automated checks
- Detailed write-up linked on the author's blog
Related event: Hands-on: Teaching a Local 4B Model New Domain Knowledge via CPT+RAG(5 posts)→
More from coding & agent
- Too many 'X times faster' claims hide extremely narrow benchmarks, says Ruff author — charliermarsh · 2026-09-25
- Ruff author Charlie Marsh: microbenchmark culture is getting worse — charliermarsh · 2026-09-25
- Claude API adds inline tool definitions for adding full tools mid-conversation — EricBuess · 2026-09-25
- Hands-on repo teaches GitHub stacked PRs with gh stack CLI and an AI coding agent workshop — DanWahlin · 2026-09-25
- Energy's AI agent can log in and pay for you, with credentials kept local and hidden from LLMs — emilahlback · 2026-09-25
- OpenRouter launches unified web search server tool with per-provider pricing — jeff_weinstein · 2026-09-25