OptiMer Wins EMNLP2026 Outstanding Paper: Post-Hoc Data Mixing With 15-35x Lower Search Cost
prajdabre · x · 2026-10-28
The NICT paper "OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training" won an EMNLP2026 Outstanding Paper Award.
- Problem: data mixture ratios in continual pre-training are expensive hyperparameters fixed before training; a bad choice can waste weeks of compute.
- Method: train one CPT model per dataset, extract each model's distribution vector (the parameter shift induced by that dataset), then search optimal composition weights post-hoc via Bayesian optimization — decoupling ratio selection from training.
- Results: on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code), OptiMer consistently beats data-mixture and model-averaging baselines with 15-35x lower search cost.
- Bonus: optimized weights can be interpreted as mixture ratios and retrained; the same vector pool can be re-optimized for new objectives without retraining.
More from Research
- Purdue's IDEAS Lab demos GPS-free drone that tracks runners through dense forests on-chip — DavidSKrueger · 2026-10-28
- Webinar to present empirical research agenda for studying inter-AI morality — dioscuri · 2026-10-28
- T-DIG diversity policies zero-shot transfer to GPT Image 2 and Gemini — simi_97k · 2026-10-28
- Cohere Labs hosts Oct 28 research session with Lila Sciences and Physical Intelligence scientists — tw_killian · 2026-10-28
- Reward hacking in T-DIG: diversity-only reward yields blank images and off-prompt sets — simi_97k · 2026-10-28
- T-DIG releases first text-conditioned diversity dataset with 20k filtered prompts — simi_97k · 2026-10-28