Experiment: continued pretraining for stable domain knowledge, RAG for dynamic data
funJS · reddit · 2026-09-16
A learning-oriented experiment combining RAG with continued pretraining (CPT) to give a model both stable domain knowledge and dynamic updates.
- Qwen 3.5 4B was fine-tuned via CPT on a fictional subway system so it learns the map and plans multi-transfer routes; the corpus was designed to avoid memorization and force generalization
- Once map knowledge stabilized, a RAG step injects dynamic info (station closures, nearby events) that CPT can't cover
- Full write-up on the author's blog
Related event: CPT Plus RAG Teaches Qwen3.5 4B a Fictional Metro System(3 posts)→
More from Research
- Tencent unveils Hyra 2.0, a research agent that builds reusable scientific tools — TencentHunyuan · 2026-11-02
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- Mathematician Tony Feng: current-paradigm LLMs lack the diversity of thought research needs — littmath · 2026-11-02
- Measure Theory vs Probability: 'You're Confusing the Finger With the Moon' — LucaAmb · 2026-11-02
- DStill Distills Multi-Step Denoising Into One Pass, 6.45x Speedup for Diffusion Speculative Decoding — LucaAmb · 2026-11-02
- Radial gate objective cuts diffusion transformer FID from 16.24 to 11.83 — Junha Park · 2026-11-02