OpenCLIP's new native ModernText encoder: more customizable than Transformers ModernBERT, decoder-capable
wightmanr · x · 2026-09-04
After OpenCLIP added a native ModernText encoder, users asked how it differs from the pretrained ModernBERT models available via Hugging Face Transformers. The author explains both share similar goals, but the native implementation offers more config knobs and easier customization, while the Transformers versions provide solid pretrained weights with more pooling constraints.
Two additional notes: ModernText extras are easy to customize (suggestions welcome), and it's not just an encoder — the additions were designed to work as encoder and/or decoder for models like MaMMUT.
Related event: OpenCLIP Maintainers Explain Native ModernText vs Transformers ModernBERT(2 posts)→
More from Research
- UCSD study: synthetic dialogues from non-conversational data bootstrap recommender systems — UCSanDiego · 2026-09-04
- New Lower Bounds Push Gradient Descent Acceleration Beyond the Nesterov Era — burny_tech · 2026-09-04
- Reef: Open-Source Infra for Self-Improving Agents Gains ~300 Stars in 2 Days — pliang279 · 2026-09-04
- Astra claims five open Erdős problems solved, unlocking a parallel compute paradigm for math — BLUECOW009 · 2026-09-04
- De novo designed antibodies protect against lethal cobra venom in vivo — BrianHie · 2026-09-04
- Roman Telescope: 100x Hubble's field of view, to catalog 1 billion galaxies in year one — PeterDiamandis · 2026-09-04