Experiment: continued pretraining for stable domain knowledge, RAG for dynamic data
funJS · reddit · 2026-09-16
A learning-oriented experiment combining RAG with continued pretraining (CPT) to give a model both stable domain knowledge and dynamic updates.
- Qwen 3.5 4B was fine-tuned via CPT on a fictional subway system so it learns the map and plans multi-transfer routes; the corpus was designed to avoid memorization and force generalization
- Once map knowledge stabilized, a RAG step injects dynamic info (station closures, nearby events) that CPT can't cover
- Full write-up on the author's blog
Related event: Experiment Combines Continued Pretraining with RAG for Subway Navigation(2 posts)→
More from Research
- Video models 'commit' to physics at a depth boundary, new paper finds — ZimingLiu11 · 2026-09-16
- KD in mid-training favors reasoning over factual recall, AI2/UW paper finds; Switch Distillation proposed — LukeZettlemoyer · 2026-09-16
- First large-scale 'AI in Science' report released as start of new research agenda — soumitrashukla9 · 2026-09-16
- MIT dataset distillation paper led by Tristan Cazenavette lands on arXiv soon, shown in artist styles — giannis_daras · 2026-09-16
- Better pretraining yields better robot policies, robust even with sparse post-training data — chris_j_paxton · 2026-09-16
- Rhoda shows web-video pretraining scaling improves real-world robot performance — chris_j_paxton · 2026-09-16