Combining RAG and Continued Pretraining on Qwen 3.5 4B: A Subway System Experiment
funJS · reddit · 2026-09-17
A Redditor ran a learning experiment: continued pretraining (CPT) on Qwen 3.5 4B taught the model a fictional subway system well enough to plan routes with multiple transfers, with careful corpus design to avoid memorization and encourage generalization.
A RAG step was then added to inject dynamic data — station closures, nearby concerts, real-time announcements — complementing the stable knowledge learned via CPT. The experiment illustrates a clear division of labor: CPT for stable domain knowledge, RAG for dynamic updates. Full write-up available on their blog.
Related event: CPT Plus RAG Teaches Qwen3.5 4B a Fictional Metro System(3 posts)→
More from coding & agent
- Tencent unveils Hyra 2.0, a research agent that builds reusable scientific tools — TencentHunyuan · 2026-11-02
- What broke when I ran a leaderless agent swarm with no orchestrator — 7 failure modes and fixes — LostHumanity67 · 2026-11-02
- Forced yes/no prompts push GPT and Claude to 95%+ confidence on Goldbach's unproved halt test — Nir777 · 2026-11-02
- Automation took over my daily work — so why am I still sitting in the office for 8 hours? — Am1rSori · 2026-11-02
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- "Coding isn't the bottleneck anymore — usage windows are," says founder Kyle Gawley — kylegawley · 2026-11-02