CrossBERT makes frozen BERT representations better and trains 2x faster
burkov · x · 2026-07-26
Meta and NYU researchers show that standard BERT-style text encoders are usually judged after fine-tuning, which makes it hard to tell whether pretraining alone produced reusable representations.
Their test freezes the encoder and trains only a small model on top. Under that setting, larger BERT models can become worse at producing directly reusable representations even as masked-word prediction improves.
To address this, CrossBERT splits the job:
- the main encoder focuses on representation building,
- a smaller predictor reconstructs masked words from those representations.
Across multiple model sizes, CrossBERT’s frozen representations improve steadily, while standard BERT’s often degrade. The paper also reports roughly 1.5–2× faster training and better data efficiency.
Related event: Meta and NYU Propose CrossBERT to Fix BERT Scaling Flaws(2 posts)→
More from Research
- Mathematician Daniel Litt Launches Problem Repo to Track Human vs AI Progress: 15 Problems, 1 Solved — littmath · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Navier-Stokes, Riemann, P vs NP: what this week's math buzzwords mean for you — koltregaskes · 2026-09-11
- Fruit fly brain as an LLM: connectome-driven language model demo goes live — ngxson · 2026-09-11
- Harry Collins: LLMs can't do frontier science because they can't invent new language — whoamisri · 2026-09-11