Meta, NYU and Yann LeCun argue BERT-style encoders break under scaling
pbaylies · x · 2026-07-25
The shared paper argues that standard BERT-style encoders have a scaling bug: as pretraining compute grows, their frozen representations can get worse rather than better.
To fix this, the authors propose CrossBERT, a two-part design that separates representation learning from token reconstruction. The paper says this enables larger masking ratios and gradient collection across all tokens, improves throughput and sample efficiency, and yields monotonic scaling on MTBE (eng, v2) and frozen GLUE benchmarks.
Related event: Meta and NYU Propose CrossBERT to Fix BERT Scaling Flaws(2 posts)→
More from Research
- Datology webinar: Curation Studio matched top data baselines with under 20% compute — josh_wills · 2026-10-27
- KANLab: an interactive browser playground for Kolmogorov-Arnold Networks — Lukas Niehaus · 2026-10-27
- CMU's LLM2Cert synthesizes stability certificates verified via SMT and Lean — CarnegieMellonU · 2026-10-27
- NVIDIA open-sources FLORA, a robotics data engine ~125x cheaper than GPT-6 Astra — nvidia · 2026-10-27
- RSNA 2026 Deep Learning Labs return: eight hands-on AI classes over four days in Chicago — PRouzrokh · 2026-10-27
- Steering two orthogonal concept neurons makes a model compose "gnome" — Sauers_ · 2026-10-27