Meta, NYU and Yann LeCun argue BERT-style encoders break under scaling
pbaylies · x · 2026-07-25
The shared paper argues that standard BERT-style encoders have a scaling bug: as pretraining compute grows, their frozen representations can get worse rather than better.
To fix this, the authors propose CrossBERT, a two-part design that separates representation learning from token reconstruction. The paper says this enables larger masking ratios and gradient collection across all tokens, improves throughput and sample efficiency, and yields monotonic scaling on MTBE (eng, v2) and frozen GLUE benchmarks.
More from Research
- Bayesian thinking left room in LLM sampling, but the field spent three years on structured generation — remilouf · 2026-07-25
- AIMACS26 talk spotlights structured LLM outputs and Lean CSLib — swarat · 2026-07-25
- Paper argues LLMs are “anthropomimetic,” mirroring human flaws as well as strengths — dioscuri · 2026-07-25
- Protein design preprint compresses sequences into 8,192 ProtWords and generates folded proteins — CatAstro_Piyush · 2026-07-25
- An AI detector correctly labels four AI essays and two human ones, then the writer jokes that only machines can tell — paul_cal · 2026-07-25
- HartwigGroup joins Genesis Mission to build ML models for synthetic chemistry — CatAstro_Piyush · 2026-07-25