Zero-cost MLM-head rescaling stabilizes SPLADE training, yet GTE-MLM remains unbeaten
tomaarsen · x · 2026-11-20
tomaarsen discusses the paper Rescaling MLM-Head for Neural Sparse Retrieval (Jang et al.). SPLADE-style learned sparse retrieval uses raw MLM-head outputs for sparse lexical matching; stronger encoders like ModernBERT and Ettin have large MLM-head L2 norms, which amplifies sparse activations and can cause performance collapse under standard SPLADE recipes. The fix: rescale the MLM-head projection by a constant factor at initialization — zero cost, no architecture or objective changes — stabilizing training and making large-norm backbones competitive on in-domain and out-of-domain benchmarks. Still, the GTE-MLM line remains the strongest even after the fix, and the author links a finetuning guide.
More from Research
- Murmur Transcribes Long Audio 2.5x Faster Than Single-Pass, Beats WhisperX — bariskasikci · 2026-11-20
- AI-designed CPU 'Frostbite' claims 4.59x perf-per-watt over silicon-proven design — davidcrawshaw · 2026-11-20
- OpenAI data: agents reshape research but experiments-per-researcher up only 1.68x — CatAstro_Piyush · 2026-11-20
- Salesforce releases BLIP-Live: SOTA proactive streaming video understanding with PAUC-F metric — Salesforce · 2026-11-20
- RICoT paper: LLMs read page streams like humans for document segmentation, boundary F1 0.915 — Prashant Khatri · 2026-11-20
- One neuron per language hits 99.1% language-ID accuracy, beating dedicated models — julianweisser · 2026-11-20