UltraBERT Rethinks BERT Pretraining, First Encoder to Hit SOTA on Both Classification and Retrieval
Jungwoo Park · hf · 2026-10-14
The UltraBERT paper systematically re-examines the pretraining recipe for bidirectional encoders, yielding three counterintuitive findings:
- Compute saturation point: downstream tasks saturate after trillions of tokens; adding more compute yields no benefit
- Masking ratio: low masking ratios favor classification while high ratios favor retrieval—overturning the established claim that larger models need higher masking ratios
- Distillation: pruning-based distillation improves classification at low masking ratios but hurts retrieval at high ones
Accordingly, UltraBERT assigns separate LM heads to high and low masking ratios within the same batch and restricts the distillation objective to the low ratio. It is the first encoder to achieve SOTA on both classification and retrieval: GLUE 91.0, BEIR 47.7. The model comes in 150M/380M sizes with 32K context, providing a systematic basis for compute allocation, masking design, and knowledge transfer in bidirectional models.
More from Research
- Chollet: Ideal AI-for-Science Outcome Is Millions of AI-Aided Scientists, Not a Few Rich Labs — fchollet · 2026-10-14
- Feeling a dipole's electric field along an ellipse: 2.08kq at the end, 0.13kq on top — TinfoilTricorn · 2026-10-14
- Alibaba's Representation Guidance: plain pixel-diffusion Transformers beat bigger models — bdsqlsz · 2026-10-14
- Visualizing the Fourier series of a sawtooth wave, with Gibbs overshoot hitting ~9% — TinfoilTricorn · 2026-10-14
- Gemma Embedding 2 hits SOTA on Vietnamese speech after 3,000 hours of audio, 2 hours on 8xH100 — stablequan · 2026-10-14
- KILVO: robust multimodal odometry for humanoid robots fusing encoders, IMU, LiDAR and vision — rsasaki0109 · 2026-10-14