Teaching local LLMs new domains: CPT + RAG experiments with full evals
funJS · reddit · 2026-09-25
A detailed writeup of experiments teaching a local model (Qwen 3.5 4B, trained with Unsloth LoRA) a new domain via continued pretraining (CPT), in four phases: (1) picking training sets that generalize to unseen questions, (2) comparing internalized knowledge (CPT) vs. RAG-injected content for reasoning, (3) combining CPT with RAG rather than treating them as competitors, and (4) a comprehensive eval strategy including SFT to force strict schema outputs for automated checks. Full findings published on the author's blog.
Related event: Hands-On: Teaching Domain Knowledge to a Local 4B Model via CPT and RAG(3 posts)→
More from coding & agent
- Pigment Agents now reads and posts in Slack, inheriting user permissions — mattturck · 2026-11-12
- Map of the agent economy: 50+ startups across payments, spend control, identity, memory — venturetwins · 2026-11-12
- OpenCode launches Browse, an on-device browser agent with local logins — Saul_Loveman · 2026-11-12
- VentureTwins maps the B2A economy: dozens of agent infra and app companies in first series installment — venturetwins · 2026-11-12
- Google releases 76-page 'Agents Companion' whitepaper on agent engineering — mdancho84 · 2026-11-12
- Spinning up containers for 20-line agent tools is backwards: Safescript compiles provable, 50ms-deploy apps — uriwa · 2026-11-12