Combining Continued Pretraining with RAG: Teaching Qwen 3.5 4B a Fictional Subway Map

funJS · reddit · 2026-09-16

The author ran an experiment using Unsloth to continue-pretrain Qwen 3.5 4B on a fictional subway system until it could plan multi-transfer routes, deliberately avoiding a memorization-heavy corpus. Once the map knowledge generalized, a RAG layer was added for dynamic data like station closures and nearby events. The combo — CPT for stable knowledge plus RAG for dynamic data — is a practical recipe for injecting new domains into small models.

Original post →

Combining Continued Pretraining with RAG: Teaching Qwen 3.5 4B a Fictional Subway Map

More from coding & agent