Scaling laws for data-constrained dense retrieval: multiplicative power law across BERT retrievers
_reachsumit · x · 2026-11-05
- This paper studies how minimum attainable evaluation loss (MAEL) scales with model size and unique fine-tuning samples for dense retrieval under data-constrained regimes where annotations are scarce and reused across epochs.
- Findings: learning-rate choices affect trajectories but barely affect final MAEL—learning curves collapse after rescaling the optimization-step axis, and CKA shows MAEL checkpoints learn highly similar representations.
- Fine-tuning 35 BERT-family retrievers on MS MARCO subsets shows MAEL follows a simple multiplicative power law over non-embedding parameters and unique samples, outperforming additive formulations.
- The authors also derive a scaling law for the training step reaching MAEL and show optimal weight decay follows an approximately invariant EMA timescale, yielding better in-domain and out-of-domain retrieval.
More from Research
- lm-eval Pitfall: Chatty Models Break Letter-Only Prompts, Regex Grabs Wrong Answers — mancubus77 · 2026-11-05
- Evolution can't copy homework: why species improve slower than iPhones — louisvarge · 2026-11-05
- Weaviate Podcast: Forgetting Is the Key to Scaling Agent Discovery, With Berkeley's Shangyin Tan — CShorten30 · 2026-11-05
- Epoch AI rebuilds its ECI with Bayesian methodology for sharper model comparisons — scaling01 · 2026-11-05
- Zyphra unveils Hierarchical Attention claiming 35x speedup at 10M tokens — teortaxesTex · 2026-11-05
- PhD student blogs on challenges of latent action models for robotics pretraining — EugeneVinitsky · 2026-11-05