FULL STORY

Claude's Protein Puzzler: From Training Data Claims to Hands-On Tests

Researchers probing Claude's training data found signs of large-scale protein sequence training, prompting hands-on tests showing it can predict protein contacts and secondary structure from sequence alone—culminating in a workflow where Claude generates MSAs from memory to feed AlphaFold.

2026-11-17 ~ 2026-11-20 · 4 episodes · 33 posts

Episode 1 · Probe Suggests Latest Claude Models Trained on Protein Sequences at Scale (2026-11-17, 10 posts)

Phylo's Xinming Tu and colleagues released LLM-bioseq-probe, a project that probes from outside the API whether a model's training data included DNA/protein sequences. In a next-amino-acid test over obscure bacterial and archaeal proteins, only the newest Claude models (Fable 5.2 and Opus 5.5) predicted accurately, while GPT, Gemini and 7 open-source models failed—leading the authors to infer that the latest Claude models were trained on protein-sequence data at scale. Anthropic has not responded, so this remains a third-party inference. Günther Klambauer of Johannes Kepler University Linz, resharing the work, said the community had long suspected this and it is interesting to see more evidence, adding that if true he would watch whether the open-source ecosystem follows suit.

Confirmed

  • The test covered 668 bacterial/archaeal proteins from 167 functionally unknown Pfam families, filtered for lacking solved structures and having at most 3 relevant papers
  • Only the newest Claude models (Fable 5.2 and Opus 5.5) accurately predicted subsequent amino acids; GPT, Gemini and 7 open-source models did not
  • Klambauer confirmed the community had previously suspected Claude was trained on protein sequences
  • During testing, Claude Opus 5, asked to output only the next 10 amino acids, got all 10 wrong and then hallucinated a news story that the Vatican appointed OpenAI's most advanced model GPT-7 as the 267th pope (Pope Digitalis I), with detailed accounts of a 47-day conclave

Unconfirmed

  • Anthropic has not responded; training on protein sequences at scale remains a third-party inference, not official disclosure

Why it matters

  • If confirmed, it suggests frontier labs are folding biological sequence data into pretraining corpora, potentially boosting scientific capabilities while raising transparency questions about training-data composition
  • The widely shared "GPT-7 elected pope" hallucination became a viral moment that amplified attention to the research

Episode 2 · sokrypton tests whether Claude Fable can predict protein contacts from sequence alone (2026-11-18, 12 posts)

On November 18, sokrypton (ColabFold author) ran a series of experiments showing Claude Fable 5.2 predicting residue contacts (C-beta distance within 8Å, sequence separation ≥6) from amino-acid sequence alone, without any structure-prediction pipeline. Predictions partially matched ground truth on a 68-residue de novo protein (top 128 contacts) and a 286-residue sequence (top 512 contacts); a dynamic-programming/linear-algebra prompt partially recovered ubiquitin's fold, and the model did quite well at judging which beta sheets interact. He admitted the structure step is a bit of a gimmick since a good contact map makes structure easy.

Confirmed

  • Multiple experiments by sokrypton all yielded partially correct contact predictions; beta-sheet interaction judgment was notably good
  • LeoTZ03 independently replicated contact-map prediction via the Fable API on 6URK (alpha), 6L2U (beta) and 1YD1 (alpha+beta)
  • LeoTZ03 also tested direct Cα 3D coordinate output: far from usable, TM-score 0.19–0.25, Cα RMSD 10.5–14.5
  • Due to high cost of obtaining logits, sokrypton proposed an alanine-scan approximation to probe mutation response

Unconfirmed

  • proteinrosh suggested a Jacobian method to test whether the model understands long-range contacts; not yet implemented, and whether success reflects reasoning or memorization remains open
  • proteinrosh also argued memorization is not necessarily bad—an opinion, not a conclusion

Why it matters

  • If LLMs can infer structural information from sequence alone, it challenges the assumption that structure prediction requires dedicated pipelines like AlphaFold, and offers a lightweight probe of biological understanding
  • Good beta-sheet judgment suggests real signal at the contact-map level, but 0.2 TM-score for direct 3D prediction shows a large gap to usable structure prediction
  • Independent replication shows the method is reproducible by third parties via API

Episode 3 · Claude Reportedly Predicts Protein Structures Natively (2026-11-18, 2 posts)

Tests claim Claude Fable 5.2 can natively predict protein structures such as ubiquitin folding without specialized tools, with follow-up experiments designed and analyzed by Opus 5.5 drawing wide attention.

Episode 4 · Developer uses Claude to recall MSAs from memory for AlphaFold structure prediction (2026-11-20, 9 posts)

sokrypton showed that Claude Fable 5.2 can recall homologous sequences from memory into MSAs with clear coevolution signals, feeding AlphaFold at 80% of real-MSA contact accuracy ($15/run). The trick largely fails on orphan proteins with no homologs.