2026-10-23
ProtTale writes function text from sequence and scores trust with pJay. On 1,031 unseen enzymes the filter lifts short-protein precision from 25.8% to 45.9%.
UniProtKB release 202602 has manual curation for 0.38% of known proteins. Homology transfer works when a close annotated neighbor exists. It cannot describe a function the reference set does not already contain, and it mislabels proteins that look alike and act differently.
Protein-to-text models write free-form descriptions that read well whether or not they are right. BLEU and ROUGE reward n-gram overlap. BERTScore is hard to pin to a biochemical claim. Mapping the text back onto Gene Ontology adds enough noise to swallow the gaps between methods. The same extractor, run on curated Swiss-Prot text, reaches a Wang similarity of only 0.67 against that entry's own molecular-function terms. The missing third is lost in the mapping.
ProtTale, from the Bromberg and Yang groups at Emory with a collaborator at Sano in Kraków, was posted to bioRxiv on 20 October 2026. Four pieces sit in series. ESM-C 300M embeds the amino-acid sequence, truncated at 1,024 residues because encoder attention memory grows with length squared. A Q-Former initialized from PubMedBERT, with cross-attention in every block, compresses the variable-length residue vectors into K=4 query vectors. Galactica-1.3B, LoRA-tuned, writes a Swiss-Prot-style function description conditioned on those vectors, capped at 128 tokens. A reliability head is the fourth piece.
Training has two stages, then a head fit on a frozen trunk.
Stage 1 is supervised by machine-inferred protein similarity, MIPS. ProtT5 embeddings of 5.2 million UniRef90 proteins that have at least 10 homologs, and 5.5 billion edges at cosine similarity above 0.77, yield 1.2 million communities. Same-community pairs are positives. Different-community pairs are negatives, meaning "not detectably close in embedding space", not "functionally different". Among 1,000 annotated same-community Swiss-Prot pairs, mean sequence identity is 37.5% (95% CI 36.5–38.6%), 58.8% have TM-score at least 0.7, and 63.1% share an EC number at the fourth level. Proximity is a usable, imperfect proxy.
The stage is a teacher-student alignment. A shared Q-Former shapes a text space from the two curated Swiss-Prot descriptions, with a binary cross-entropy on temperature-scaled cosine similarity and the MIPS label as target. Both sequences pass through ESM-C and the same Q-Former. Pointwise and pairwise losses pull the sequence side toward the text side. A stop-gradient keeps those losses from updating the Q-Former. The pairing comes from embedding geometry. The text is still curated. Proteins longer than 1,200 residues, or in UniRef90 clusters with fewer than 10 members, contribute no Stage 1 pairs.
Stage 2 freezes that encoder and trains the Q-Former plus LoRA on Galactica with teacher-forced autoregressive cross-entropy. The training split has 207,197 proteins, capped at 10 per CD-HIT cluster at 40% identity. Validation and test are 1,000 proteins each. The unseen set has 1,031 proteins from clusters that entered Swiss-Prot after 2022, with no 40% identity neighbor in the training clusters.
The reliability head never reads the generated text. It concatenates the encoder output mean-pooled over valid residues and the Q-Former output mean-pooled over queries, then a dropout and one linear layer. The sigmoid output, pJay, is the predicted probability that the judge will call ProtTale's description a Yes. Yes is the positive class. No, Maybe and Vague are collapsed to negative. Labels come from the validation and test splits. The training split is skipped because the generator is likely overfit there. Gradients do not flow back into the encoder or the Q-Former. The cutoff is calibrated, not chosen by hand. Residue-shuffled copies of the unseen sequences never reach the long right tail of real pJay scores. A Fisher exact sweep over 401 thresholds puts the sharpest separation at 0.537. Every discrete filter in the paper is pJay > 0.537.
The judge is Claude Opus 4.7, prompted to score molecular function only, on four labels: Yes, No, Maybe, and Vague. Vague penalizes generic phrases such as "binding protein". Biochemically adjacent activities are not treated as the same, so ATP binding is not kinase activity. Five independent runs are taken, and a prediction counts as correct only if at least three say Yes. Run-to-run agreement is 75–81%, with Cohen's κ of 0.61 to 0.73.
Every unseen protein carries an EC number, so the benchmark is enzyme function. 47% have no homolog at 40% identity anywhere in Swiss-Prot. At least one method is correct on 450 of 1,031 proteins (44%). The other 581 are missed by everyone. Four leading methods land at Yes rates of 24.9–26.5%. Protein2Text is at 6.2%. All six pairwise McNemar tests among the four leaders give Holm-adjusted P = 1.00. Accuracy is a tie.
The tie is not a shared set of proteins. In the three-method overlap, 109 proteins are correct for all three, ProtTale alone gets 79, Prot2Text 59 and ProtT3 34. In the five-method comparison, only 17 proteins are correct for every method, and ProtTale's unique set of 66 is one larger than the other four unique sets combined (65). Pairwise Cohen's κ on the four labels is mostly 0.10–0.34. The exception is Prot2Text versus ESM2Text, which share a backbone, at 0.60.
The biases differ too. Whether ESM2Text and Prot2Text are right tracks sequence-family size and the best TM-score against the PDB. ProtT3 tracks how many PDB neighbors have TM-score at least 0.7, and larger structural families go with worse function calls. ProtTale's hits are not explained by those sequence or structure signals.
GO extraction flattens the difference. Wang similarity without ancestor expansion is 0.67 for curated text after extraction, 0.556 for Prot2Text, 0.561 for ProtTale, 0.569 for ESM2Text, 0.605 for ProtT3 and 0.443 for Protein2Text. Empty extractions run from 10% (ProtT3) to 32% (ESM2Text). The spread among the four leaders is about one seventh of the third of the signal lost by mapping curated text onto GO. Some empty calls are Claude content-safety refusals, for example on resistance-enzyme sentences, and they are counted as empty for every method.
pJay > 0.537 keeps 201 of 1,031 proteins (19.5%). Because the score reads sequence only, every method is filtered to the same proteins: 153 of 852 short proteins (at most 1,024 residues) and 48 of 179 long ones. On short proteins all four leaders gain. ProtTale moves from 25.8% to 45.9% precision, and from 0.56 to 0.61 Wang similarity. ProtT3 reaches 46.4% precision and 0.64 Wang, McNemar P = 0.64 against ProtTale. ESM2Text moves from 26.2% to 40.1%, Prot2Text from 27.4% to 39.9%. Protein2Text moves from 6.7% to 9.3%, which the abstract's "nearly twofold for every method" does not cover. On long proteins ProtT3 does not gain (14.4% to 14.2%). Filtered ProtTale is at 32.1%, on 48 proteins, too few for a stable claim.
A moving threshold trades coverage for precision. At pJay > 0.75, 20 of 1,031 predictions remain and ProtTale reaches 66%, ProtT3 61%, and the other two leaders about 50%. Precision near 90% is two predictions. On short bacterial proteins ProtTale rises from 19.2% to 51.2%, the largest gain the paper reports, with Benjamini-Hochberg P = 5.7 × 10−6.
Dropping Stage 1, and initializing the frozen encoder from the original ESM-C weights, cuts the short-protein Yes rate from 25.8% to 12.7% (McNemar P = 7.7 × 10−16). Long proteins barely move, 29.8% versus 29.1%, P = 0.60. Proteins that long contributed almost no Stage 1 pairs, so the ablation matches how the pairs were built.
An author who did not build the model scored 232 unseen ProtTale descriptions against Swiss-Prot on a 1-to-5 scale, blind to pJay and to the judge. Sampling took the tails, pJay at most 0.3 or at least 0.7, and dropped cases where the reference was more than twice as long as the prediction. 209 of the 232 were predictions the model itself did not trust, against about 40% of the full unseen set. Treating curator scores of 4–5 as agreement, the judge has 64% precision, 72% recall and 86% accuracy (Fisher P = 6.3 × 10−17, Spearman ρ = 0.585). Every pair the curator called equivalent (5) was a judge Yes. Of 56 pairs scored unrelated (1), the judge marked 55 as not Yes. The most common score is 3 (82 of 232, 35%): right functional neighborhood, wrong specific role. For Drosophila adenylate kinase Q8IQG9, Swiss-Prot describes reversible phosphate transfer between ATP and AMP (EC 2.7.4.3). ProtTale wrote phosphorylation of pyrimidine nucleoside monophosphates (EC 2.7.4.14). Both are nucleotide kinases. The judge correctly said No.
Above pJay 0.70, 23 curated predictions, 57% score 4–5 and 13% score 1–2. Below 0.20, 60 predictions, 68% score 1–2 and 8% score 4–5. In every bin the judge's Yes rate stays within five points of the curator's 4–5 share. The head never saw the text. The curator never saw the score.
Another two points of average Yes rate still does not tell an experimentalist which annotation to build an assay around. pJay reads sequence only, so the same cutoff filters ESM2Text, Prot2Text, ProtT3 and Protein2Text. On short proteins the four leaders move from about 26–27% to 40–46%, at a coverage of about one fifth. That is a list for wet-lab triage, not a whole-database annotator.
The MIPS alignment is why ProtTale ties the field. Without it, short-protein precision is cut in half, and ProtTale is the method with the largest unique correct set. Inference is about 7,700 sequences per GPU-hour. The 1,031 unseen proteins took eight minutes on one RTX 6000 Ada. The head is a mean pool and one linear layer.
The benchmark is mostly enzymes with an EC number. Metagenomic and de novo designed sequences are untested. The prose was learned from Swiss-Prot, so free text does not escape that vocabulary.
The judge is one prompt on Claude Opus 4.7. Five runs measure sampling noise, not version drift. Labels and the prompt are released, and a new judge means a rescore. The authors also did not benchmark mean token log-probability, the obvious decoder-side confidence, so it is open whether pJay beats that cheaper signal.
The curator is a coauthor. Sampling covers only the tails, and the band from 0.30 to 0.70, which holds 57% of the unseen set, has no human scores. The 57% "worth acting on" figure above 0.70 rests on 23 proteins. Precision near 90% is two predictions and is not an operating point.
Generation stops at 128 tokens. 233 of 1,031 unseen predictions (22.6%) hit the cap without an end-of-sequence token. The curator did not score truncated text lower (Mann-Whitney P = 0.25) in this sample of 232. Stage 1 validation sequence AUC is about 0.87. The head's positive-class F1 is about 0.55, against 0.476 for always predicting the positive class. The filter's precision gain comes from discarding about 80% of proteins.