UltraBERT beats ModernBERT and Ettin with GLUE 91.0 and BEIR 47.7, 32K context, open weights
kalyan_kpl · x · 2026-10-14
UltraBERT, a new open-source BERT-family encoder, claims SOTA on both classification and retrieval: 91.0 on GLUE and 47.7 on BEIR, outperforming Ettin BERT, ModernBERT, BERT and RoBERTa.
Key details:
- Dual masking ratios (5% for classification, 50% for retrieval) with two separate LM heads
- 32K token context — 4x ModernBERT
- Two sizes: 150M and 380M, distilled from a 1.3B teacher
- Trained on 2T+ tokens; weights available on Hugging Face
A strong new default candidate for embedding and retrieval workloads.
Related event: UltraBERT: Small Open Encoder Hits SOTA on Classification and Retrieval(3 posts)→
More from Models
- Step 5 Preview architecture blog details Sparse GQA with block-wise token merging cutting indexer cost 8x — teortaxesTex · 2026-10-15
- Fish Audio Drama 3 lands in vLLM-Omni: 128 concurrent streams on one H200 — vllm_project · 2026-10-15
- Why is GPT-6 Astra ranked so low on LM Arena? Outscored by GPT-5.6 and 6.1 — AreaExact7824 · 2026-10-15
- Stance test heat map shows Sonnet 5.5 resists framing manipulation other models succumb to — teortaxesTex · 2026-10-15
- Argon (Medium) burned 85% of a 5-hour usage limit in just 5 minutes — Avalunne · 2026-10-15
- Score vs cost chart puts Gemini 4 Argon on the Pareto frontier, Kimi K3 far off it — Hesamation · 2026-10-15