Combining RAG and Continued Pretraining on Qwen 3.5 4B: A Subway System Experiment
funJS · reddit · 2026-09-17
A Redditor ran a learning experiment: continued pretraining (CPT) on Qwen 3.5 4B taught the model a fictional subway system well enough to plan routes with multiple transfers, with careful corpus design to avoid memorization and encourage generalization.
A RAG step was then added to inject dynamic data — station closures, nearby concerts, real-time announcements — complementing the stable knowledge learned via CPT. The experiment illustrates a clear division of labor: CPT for stable domain knowledge, RAG for dynamic updates. Full write-up available on their blog.
Related event: CPT Plus RAG Teaches Qwen3.5 4B a Fictional Metro System(3 posts)→
More from coding & agent
- The harness tax: Claude Code costs 2x Pi at the same 15.3 turns, with 10x initial context on SWE-bench Lite — HamelHusain · 2026-09-17
- Jev: an open-source action-picker that splits agent thinking from clicking — alexcovo_eth · 2026-09-17
- CROA open-sources a deterministic execution layer enforcing trajectory-level constraints on AI agents — CROA_PROJECT · 2026-09-17
- Moving coding agent memory off-machine: from Dropbox CONTEXT.md to MCP shared memory — Asly97 · 2026-09-17
- What's the worst 'fix' you've seen an AI coding agent make? — fromkrish · 2026-09-17
- Self-taught builder shipped 900 PRs in 3 months running 18 terminal agents to automate insurance brokerage — CShorten30 · 2026-09-17