Continued pretraining vs RAG on Qwen 3.5 4B: a hands-on accuracy and performance comparison
funJS · reddit · 2026-09-12
The author ran a controlled experiment comparing a Qwen 3.5 4B model fine-tuned with continued pretraining via Unsloth against a RAG setup on the base model, measuring accuracy and inference performance to quantify the benefit of internalizing knowledge versus on-the-fly retrieval. Full write-up linked in the post.
Related event: CPT vs RAG: Testing Knowledge Internalization on Qwen 3.5 4B(3 posts)→
More from Research
- Continued Pretraining vs RAG: Hands-on Comparison on a Qwen 4B Model — funJS · 2026-09-12
- LeVJEPA: video pretraining at 5.6-20.8x less compute, matching V-JEPA 2 — Cohere · 2026-09-12
- Researcher speculates spatial reasoning leap comes from Blender training data — yoavartzi · 2026-09-12
- tszzl pushes back on Fermi paper: 50-OOM lognormal abiogenesis prior under-justified — tszzl · 2026-09-12
- OpenAI launches GPT-Rosalind for biological reasoning in API and Codex — OpenAIDevs · 2026-09-12
- MIT quantum lab uses GPT-5.6 Sol and Codex to automate chip measurements — OpenAI · 2026-09-12