Teaching local LLMs new domains: CPT + RAG experiments with full evals
funJS · reddit · 2026-09-25
A detailed writeup of experiments teaching a local model (Qwen 3.5 4B, trained with Unsloth LoRA) a new domain via continued pretraining (CPT), in four phases: (1) picking training sets that generalize to unseen questions, (2) comparing internalized knowledge (CPT) vs. RAG-injected content for reasoning, (3) combining CPT with RAG rather than treating them as competitors, and (4) a comprehensive eval strategy including SFT to force strict schema outputs for automated checks. Full findings published on the author's blog.
Related event: Hands-on: Teaching a Local 4B Model New Domain Knowledge via CPT+RAG(5 posts)→
More from coding & agent
- Open-source AI-SQL engine Quail hits 1B+ input tokens/min on a single H100 — sh_reya · 2026-09-25
- SkillRL (NeurIPS 2026): 7B model beats GPT-4o by 41% via recursive skill evolution — cihangxie · 2026-09-25
- Weave Code Max offers $50+ of coding model usage for $10/month — ycombinator · 2026-09-25
- Mad science: Jev autopilot lands a plane in a terminal flight simulator — bilawalsidhu · 2026-09-25
- Greptile reviewed 395K PRs at NVIDIA, cutting merge time from 24h to 6h — ycombinator · 2026-09-25
- Open-source Claude skill turns one prompt into stop-motion claymation films in Blender — angrypenguinPNG · 2026-09-25