Combining Continued Pretraining with RAG: Teaching Qwen 3.5 4B a Fictional Subway Map
funJS · reddit · 2026-09-16
The author ran an experiment using Unsloth to continue-pretrain Qwen 3.5 4B on a fictional subway system until it could plan multi-transfer routes, deliberately avoiding a memorization-heavy corpus. Once the map knowledge generalized, a RAG layer was added for dynamic data like station closures and nearby events. The combo — CPT for stable knowledge plus RAG for dynamic data — is a practical recipe for injecting new domains into small models.
Related event: Experiment Combines Continued Pretraining with RAG for Subway Navigation(2 posts)→
More from coding & agent
- OpenAI launches Data agent in ChatGPT Work to query company data and build dashboards — gdb · 2026-09-16
- Muse agent renews passport and books flights end-to-end in viral demo — armand_ruiz · 2026-09-16
- Bitsec's multi-model agent stack found 160+ exploits, beating a single 'superhuman' model — markjeffrey · 2026-09-16
- Dev slams agent sandbox pricing as 20x+ the cost of a $6/month always-on VPS — Aryvyo · 2026-09-16
- Abacus.AI says Smaug Flash fixes open-source models' tool-call hangs in production — bindureddy · 2026-09-16
- Dev lets AI agents reverse engineer Zapier and build his own automation app — NathanWilbanks_ · 2026-09-16