CrossBERT makes frozen BERT representations better and trains 2x faster
burkov · x · 2026-07-26
Meta and NYU researchers show that standard BERT-style text encoders are usually judged after fine-tuning, which makes it hard to tell whether pretraining alone produced reusable representations.
Their test freezes the encoder and trains only a small model on top. Under that setting, larger BERT models can become worse at producing directly reusable representations even as masked-word prediction improves.
To address this, CrossBERT splits the job:
- the main encoder focuses on representation building,
- a smaller predictor reconstructs masked words from those representations.
Across multiple model sizes, CrossBERT’s frozen representations improve steadily, while standard BERT’s often degrade. The paper also reports roughly 1.5–2× faster training and better data efficiency.
Related event: Meta and NYU Propose CrossBERT to Fix BERT Scaling Flaws(2 posts)→
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27