COLM 2026 Paper: Literature vs. STEM Data Influence Shifts Across LLM Pretraining Stages
SinclairWang1 · x · 2026-08-17
This paper accepted to #COLM2026 proposes a task-agnostic influence measure to analyze training data impact throughout language model pretraining.
Key findings include:
- Literature data matters more early in the training process.
- STEM (Science, Technology, Engineering, Math) data becomes more influential in later stages.
This reveals a shift in dependency on different knowledge domains across pretraining phases.
More from Research
- LittleLearner: 5B-Parameter Model and Corpus for Controlled Knowledge Acquisition Study — ELLIS-Institute-Tuebingen · 2026-08-17
- ByteDance & Tsinghua Release CUDA-Agent: RL-Based High-Performance Kernel Generation — AIFlow_ML · 2026-08-17
- Google paper: models hallucinate facts they already know — dejanseo · 2026-08-17
- First full 11-point humanoid robot ping-pong match powered by SMASH 2.0 — 量子位 · 2026-08-17
- 7 Free YouTube Channels to Learn LLMs, From Transformer Fundamentals to Real Apps — goyalshaliniuk · 2026-08-17
- Resource: Statistical Methods in Generative AI — RexDouglass · 2026-08-17