Bigger Isn't Always Better for Protein Language Models

bravo_abad · x · 2026-07-14

This article discusses the finding that protein language models are not necessarily better when they are bigger.

Key findings include:

In predicting protein mutation effects, smaller or medium-sized models can actually outperform larger ones. For example, ESM2-650M/3B, xTrimoPGLM-3B, and ProGen3-1B beat their larger counterparts on certain tasks.
The reason isn't just "lack of data," but because the model's output likelihood mixes in factors unrelated to true fitness, such as the number of homologous sequences in the training set, model size, and random seeds.
Due to probability normalization, large models might make wild-type residues too certain, leading to extreme mutation scores, whereas smaller models might flatten the differences.

The author reports that across 154 ProteinGYM tests, performance follows a near-bell curve relative to wild-type likelihood, peaking around 0.3. This pattern holds true for sequence-only, structure-mixed, autoregressive, and inverse-folding models.

For teams working on variant interpretation, antibody engineering, or enzyme optimization, the conclusion is clear: don't just look at parameter count when choosing a model; ensure it has actually learned the target protein family. The article also suggests a cheap pre-check: correlate the model's per-residue likelihood with a simple alignment-frequency model—a correlation of around 0.5 serves as a good working threshold before proceeding to wet lab experiments.

Original post →

Bigger Isn't Always Better for Protein Language Models

More from Research