UltraBERT: open 150M/380M encoders with 32K context and an unusual dual-masking recipe
algo_diver · x · 2026-10-15
dmis-lab released UltraBERT, Apache 2.0 encoders at 150M/380M with 32K context, built for classification, NER, QA, reward modeling and retrieval; models and paper are on Hugging Face.
The training recipe is the highlight:
- 5% masking for classification vs 50% for retrieval, with a separate LM head for each regime
- A 1.3B teacher was trained, width-pruned to 380M/150M, then distilled — logit distillation applied only to the 5%-masked sequences
- Lessons learned: encoder performance saturates early even as loss keeps falling; multi-stage data mixtures barely matter after fine-tuning; 32K context only needed in the final 100B of 2T tokens; distillation isn't automatically useful across capabilities
Related event: UltraBERT: Small Open Encoder Hits SOTA on Classification and Retrieval(3 posts)→
More from Models
- StepFun releases Step-5-Preview: 600B/27B-active flagship with 1M context — ResearchCrafty1804 · 2026-10-15
- DeepSeek V4.1 measures its own decode speed at 430-460 tok/s — then the claim falls apart — teortaxesTex · 2026-10-15
- OpenCode offers $60 of Gemini 4 Argon usage for $10 until Nov 30 — m4rkmc · 2026-10-15
- After losing 5 Claude accounts, one user's controlled tests show what actually triggers bans — jiayuan_jy · 2026-10-15
- Dev Says Jev Could Easily Detect X Tweet Spam, But X Hasn't Prioritized It — ivan_bezdomny · 2026-10-15
- DeepSeek outage ends, sparking new-model rumors that the poster himself retracts as false alarm — teortaxesTex · 2026-10-15