OpenCLIP adds layer-wise LR decay for text and image encoders to curb forgetting under distribution shift
wightmanr · x · 2026-11-13
An OpenCLIP contributor announced support for layer-wise LR decay on text and image encoders, aimed at mitigating forgetting when adapting models under large distribution shifts.
The author also notes that further fine-tuning CLIP models with contrastive losses usually fails due to data scarcity or quality, especially when data diverges significantly from the training distribution. He suggests most decent coding agents can orchestrate the needed experiments on OpenCLIP, particularly the 4.x main branch, which cleaned up the training design.
Related event: OpenCLIP Adds Layer-Wise LR Decay to Ease CLIP Fine-Tuning(2 posts)→
More from Research
- DynaDFlash: diffusion drafter boosts speculative decoding acceptance length by 5% — charles_irl · 2026-11-13
- Waterloo authors' textbook 'Elements of Dimensionality Reduction and Manifold Learning' has 392 citations — FrnkNlsn · 2026-11-13
- mcbench-lite open-sourced: agents build in Minecraft, runs locally on your keys — adonis_singh · 2026-11-13
- AI-derived ClinicalAge from EHRs correlates with cancer and chronic disease risk — EricTopol · 2026-11-13
- TRUST framework cuts bad self-improvement accepts from 45.6% to 2.78% — dair_ai · 2026-11-13
- Weight normalization emerges as new standard: neuron norms hit almost exactly 1 in nGPT-style models — SonglinYang4 · 2026-11-13