Zero-cost MLM-head rescaling stabilizes SPLADE training, yet GTE-MLM remains unbeaten

tomaarsen · x · 2026-11-20

tomaarsen discusses the paper Rescaling MLM-Head for Neural Sparse Retrieval (Jang et al.). SPLADE-style learned sparse retrieval uses raw MLM-head outputs for sparse lexical matching; stronger encoders like ModernBERT and Ettin have large MLM-head L2 norms, which amplifies sparse activations and can cause performance collapse under standard SPLADE recipes. The fix: rescale the MLM-head projection by a constant factor at initialization — zero cost, no architecture or objective changes — stabilizing training and making large-norm backbones competitive on in-domain and out-of-domain benchmarks. Still, the GTE-MLM line remains the strongest even after the fix, and the author links a finetuning guide.

Original post →

More from Research

Research channel →