Meta and NYU Propose CrossBERT to Fix BERT Scaling Flaws

Research from Meta and NYU reveals that traditional BERT-style encoders have scaling flaws, where more pre-training compute degrades frozen representations for downstream tasks. To address this, the researchers introduced CrossBERT, which improves frozen representation performance and doubles training speed.

2026-07-25 ~ 2026-07-26 · 2 related posts

• Meta, NYU and Yann LeCun argue BERT-style encoders break under scaling — pbaylies · 2026-07-25
• CrossBERT makes frozen BERT representations better and trains 2x faster — burkov · 2026-07-26