EMNLP Outstanding Paper OptiMer: data-mix search 15-35x faster than retraining
机器之心 · wechat · 2026-11-15
A team led by Haiyue Song (NICT, Japan) won an EMNLP 2026 Outstanding Paper Award (22 of 17,661 submissions) for OptiMer, which eliminates costly trial-and-error in continual pre-training data mixing.
Method: Continually pre-train the base model separately on each dataset (e.g., Japanese, Chinese, math), extract each parameter delta as a "distribution vector," then use Bayesian optimization to search optimal merging weights—each trial only requires model merging and evaluation, no retraining.
Results: 15-35x speedup in mix optimization (one traditional mix takes 129.3 hours to train; OptiMer tried 100 mixes in 8.5 hours). Average scores on C-Eval, MMLU, GSM8K, HumanEval etc. reached 70.37 vs 63.71 for traditional data mixing; the learned weights can also guide re-training, lifting scores to 68.66.
Flexibility: Distribution vectors can be reused to quickly produce domain/language-specialized models—on Gemma-SEA-LION-v4-27B, Japanese scores rose from 66.34 to 74.40 while preserving Southeast Asian language ability. Cosine analysis (vectors 0.3 similarity, near-orthogonal to instruction vector) explains mergeability; negative weights hint at machine unlearning potential.
More from Research
- Curated 10-resource path to learn LLM distributed training from scratch — East-Muffin-6472 · 2026-11-15
- Adding random noise during training improves alignment generalization, paper claims — Sauers_ · 2026-11-15
- Broad/MGH preprint models inherited variation effects on disease trajectories using longitudinal EHRs — anshulkundaje · 2026-11-15
- Llama 405B Base Model Natively Mimics o1's Chain-of-Thought from 128 Pages of Context — Sauers_ · 2026-11-15
- Magic claims >10x more efficient pretraining, matching DeepSeek V4 Base for ~$0.5M — oran_ge · 2026-11-15
- OpenDS-World benchmark: best of 12 frontier models solves only 39% of real-web deep research tasks — jiqizhixin · 2026-11-15