Teaching a Local 4B LLM a New Domain: CPT vs RAG Experiments With Evals
funJS · reddit · 2026-09-25
- Author documents a four-phase project teaching a local Qwen 3.5 4B model a new domain via continued pretraining (CPT), using Unsloth for both CPT and SFT LoRA training
- Phase 1: picking a training set that generalizes to unseen questions
- Phase 2: comparing reasoning via internalized CPT knowledge vs RAG-injected content
- Phase 3: a practical hybrid combining CPT-trained knowledge with RAG instead of treating them as rivals
- Phase 4: a full eval strategy, including SFT fine-tuning the model to output strict schemas to simplify automated checks
- Detailed write-up linked on the author's blog
Related event: Hands-On: Teaching Domain Knowledge to a Local 4B Model via CPT and RAG(3 posts)→
More from coding & agent
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11
- Why inference speed will be AI products' biggest competitive edge in 1–2 years — itsOmSarraf_ · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- OpenAI engineer runs Codex for 26 days straight hunting Atlantis, spawning "vibe archaeology" — Polymarket · 2026-11-11
- He replaced a $5,000/month ghostwriter with 11 Claude Skills and earned $47,000 from one post — aryanXmahajan · 2026-11-11
- Refero Styles ships 2,000+ DESIGN.md files to give Claude Code real design taste — bbssppllvv · 2026-11-11