2026-08-02
Protein language models under-score function-changing mutations and design novel variants worse than random; subtracting a conservation model lifts enrichment to 4-fold.
Directed evolution tries to remake a protein to do something it does not do naturally: an enzyme that acts on a different substrate, a transcription factor that binds different DNA, a transporter that moves a different ion. The space of possible amino-acid sequences is astronomically large and experiments sample only a sliver of it, so practitioners increasingly lean on machine learning to pick which mutations are worth testing.
The dominant tools are unsupervised protein fitness models: train on known natural sequences so the model captures what sequences look like in nature, then score mutations by how well they fit that distribution. Protein language models (ESM-1v and its kin) are the most hyped branch, and a good deal of recent work assumes they have learned some generalized notion of protein fitness and can therefore guide directed evolution toward functions distinct from the protein's native one.
That assumption had never been tested systematically. Existing benchmarks like ProteinGym measure how well a model predicts the effect of mutations on a protein's native function, not on a changed function. Berry, Gaudet, and Marks at Harvard built this paper to close that gap.
The authors assembled SpecificityStudio, a companion database to ProteinGym. Its core is eight multiplexed functional-assay datasets, each profiling the same set of variants against several substrates or ligands: a binding domain (PSD95-PDZ3), three enzymes (TEM-1, AmiE, PafA), two transcription factors (Pho4, RamR), and two transporters (NorA, DraNramp). Each mutation is sorted into three classes: inactive, native specificity, and altered specificity.
They then scored every variant with fourteen zero-shot fitness models spanning six classes: a site-independent conservation model (PSSM), alignment-covariation models (EVmutation, EVE, GEMME), single-sequence protein language models (ESM-1v, ESM-2, ProGen2, Tranception), hybrids (MSA Transformer, PoET, SaProt), and inverse-folding models (ESM-IF1, ProteinMPNN).
The bias metric is the altered-specificity index (ASI): it normalizes a model's average score for altered-specificity variants between its scores for native-specificity and inactive variants. ASI of 1 means the model does not discriminate; ASI of 0 means it treats altered-specificity variants as if they were inactive. The authors also simulated designing the top 5% of variants by model score and measured how many more altered-specificity variants that yields than random sampling.
The decisive move comes last. Because different models score very differently, the differences themselves may encode specificity information. The authors fit a logistic regression that weights two models' scores, and the optimal weight turns out to be negative or zero. The right operation is a weighted difference, not a sum.
The unsurprising part first: every model can separate inactive from native-specificity variants, which is their core task. ProteinMPNN was excluded for performing poorly on nearly all datasets.
The finding is on altered specificity. ASI shows that the more a model relies on sequence context, the more it under-scores altered-specificity variants. Protein language models and deep covariation models (ESM-1v, ESM-2, ProGen2, Tranception, EVE, MSA Transformer) have the lowest ASI, treating altered-specificity variants almost like inactive ones. Only the PSSM, which scores each column independently with no context, sits near ASI 1 and treats native and altered specificity alike. The inverse-folding model ESM-IF1 behaves similarly to the PSSM. Hybrids (PoET, SaProt, ESCOTT) and shallow covariation models (EVmutation, GEMME) land in between.
The bias shows up directly in design:
| Model class | Altered-specificity enrichment vs random |
| Protein language models (ESM-1v, ProGen2, Tranception) | Zero altered-specificity variants in the top 5% on several datasets |
| Deep covariation models (EVE, GEMME, ESCOTT) | Also zero on several datasets |
| Inverse-folding models | Slightly above random, but unstable |
| PSSM (site-independent) | Varies: above 4-fold on RamR, far below random on DraNramp; near-neutral on average |
No single model reaches even a two-fold average enrichment. Take the most popular protein language models and point them at an altered-function evolution campaign, and you will likely do worse than picking mutations at random.
The weighted difference does better:
| Approach | Top-5% altered-specificity enrichment |
| Any single model | < 2-fold |
| Weighted difference of PSSM and ESM-1v (all datasets) | 2.5-fold |
| Fit only on Class 1 (new non-native substrates), leave-one-out | 4-fold |
The biggest gain is on DraNramp, the only case where the wildtype has no measurable activity on the non-native substrate. The authors conclude that differences between models are most useful for designing large shifts away from the wildtype's function, less so for fine-tuning relative preference among existing substrates.
This is a measurement warning, not a new model. It punctures a widely repeated assumption that protein language models learn generalized protein fitness and so can guide evolution toward entirely new functions. The reality runs the other way: they learn the natural sequence distribution itself, which is a biased, non-equilibrium sample left behind by natural evolution. The more context a model uses and the more deeply it fits that distribution, the harder it pushes back against function regions that evolution never sampled.
Two practical consequences. First, a model that scores well on ProteinGym and predicts native function accurately does not predict altered function accurately; the two are often inversely related. For engineering a new function, a site-independent conservation model is frequently more reliable than the fanciest language model, while language models are better suited to amplifying a function the protein is already selected for. Second, because the bias is systematic, it can be exploited: the differences between model scores encode specificity information. The authors' analogy is that a student who gets every multiple-choice question wrong must have known some of the answers. Subtracting ESM-1v's score flips where it knows not to go into where to go.
The limitation the authors stress most is size and heterogeneity: only eight datasets, because most mutational-scanning studies measure a single substrate. They acknowledge the weighted difference is a floor, not a ceiling, and that larger quantitative assays will raise it; two such studies appeared while the paper was in preparation.
A few gaps the paper does not fully close. All models are zero-shot, conditioning only on sequence and at most a static structure; none uses which ligand or substrate is actually being tested. Predicting a specific new activity rather than merely enriching a library requires feeding the ligand into the model, which this work does not do. The altered-specificity labels are defined differently across datasets (linear-regression outliers in some, a global-epistasis model in others), and the conclusions are heterogeneous; the authors concede the Class 2 cases show no clear trend. Antibodies, whose evolution is more random, were excluded, so the findings do not transfer cleanly to antibody-antigen design, which is precisely the setting where ESM-class models have been reported to design new function successfully. The authors attribute the difference to antibodies' more random evolution but run no direct antibody comparison here.