UltraBERT: Small Open Encoder Hits SOTA on Classification and Retrieval

dmis-lab released UltraBERT (150M/380M, Apache 2.0), a 32K-context open encoder that achieves SOTA on both classification and retrieval (GLUE 91.0, BEIR 47.7), with the paper finding downstream tasks saturate after trillions of tokens.

2026-10-14 ~ 2026-10-15 · 3 related posts

• UltraBERT Rethinks BERT Pretraining, First Encoder to Hit SOTA on Both Classification and Retrieval — Jungwoo Park · 2026-10-14
• UltraBERT beats ModernBERT and Ettin with GLUE 91.0 and BEIR 47.7, 32K context, open weights — kalyan_kpl · 2026-10-14
• UltraBERT: open 150M/380M encoders with 32K context and an unusual dual-masking recipe — algo_diver · 2026-10-15