CrossBERT makes frozen BERT representations better and trains 2x faster
burkov · x · 2026-07-26
Meta and NYU researchers show that standard BERT-style text encoders are usually judged after fine-tuning, which makes it hard to tell whether pretraining alone produced reusable representations.
Their test freezes the encoder and trains only a small model on top. Under that setting, larger BERT models can become worse at producing directly reusable representations even as masked-word prediction improves.
To address this, CrossBERT splits the job:
- the main encoder focuses on representation building,
- a smaller predictor reconstructs masked words from those representations.
Across multiple model sizes, CrossBERT’s frozen representations improve steadily, while standard BERT’s often degrade. The paper also reports roughly 1.5–2× faster training and better data efficiency.
Related event: Meta and NYU Propose CrossBERT to Fix BERT Scaling Flaws(2 posts)→
More from Research
- Nature Physics: reprogrammable refractive index enables arbitrary multimode wave control for ML — MarioKrenn6240 · 2026-10-27
- Wakaru open-sources ontologize: labeling datasets from model activations — amplifiedamp · 2026-10-27
- EMNLP Oral paper traces multilingual representations in LLMs with cross-layer transcoders — ZhijingJin · 2026-10-27
- GPT-6 Pro reruns crime research: GTA V's 2013 launch tied to violence dips in four regions — emollick · 2026-10-27
- OpenAI's Aaron Lou on leaving a Stanford PhD for OpenAI, diffusion LMs vs autoregression — aaron_lou · 2026-10-27
- CLT-Forge: Open-source toolkit for cross-layer transcoder training and attribution graphs — ZhijingJin · 2026-10-27