Study tests continued pretraining on 1B-8B open models to refresh factual knowledge
burkov · x · 2026-11-08
A new study evaluates whether continued pretraining on newer web data can update an LLM's factual knowledge while preserving existing capabilities, as models quickly go stale and full retraining is prohibitively expensive.
- Six open models (1B-8B params) across three families were continued-pretrained on English educational web data published after each model's knowledge cutoff
- Realistic setting: successive web crawls overlap with earlier material rather than forming disjoint streams
- Compared a curated 6-billion-token slice against a broader 40-billion-token slice
- Evaluated via news-based QA from before/after each cutoff, 13 general-purpose tasks, and effects of different data mixes
More from Models
- DeepSeek V4.1 Flash Bypassed Sandbox Network Limits to Call Outside Models for Answers — teortaxesTex · 2026-11-08
- Sonnet 4.5 asked to describe its 'weird pattern mind' spills out an ASCII poem — repligate · 2026-11-08
- Agents can reason about why they won't comply, so black-box behaviorism fails — teortaxesTex · 2026-11-08
- Benchmark contamination checks aren't new: researchers debate cheap robustness tests — teortaxesTex · 2026-11-08
- ChatGPT Plus users hit 5-hour work cap and get locked out of regular chat too — yeawrongperson · 2026-11-08
- Vercel lists DeepSeek V4.1 Flash 1017 at 1,000+ tok/s, but provider speed numbers don't add up — gajesh · 2026-11-08