FULL STORY
Claude's Protein Puzzler: From Training Data Claims to Hands-On Tests
Researchers probing Claude's training data found signs of large-scale protein sequence training, prompting hands-on tests showing it can predict protein contacts and secondary structure from sequence alone—culminating in a workflow where Claude generates MSAs from memory to feed AlphaFold.
2026-11-17 ~ 2026-11-20 · 4 episodes · 33 posts
Episode 1 · Probe Suggests Latest Claude Models Trained on Protein Sequences at Scale (2026-11-17, 10 posts)
Phylo's Xinming Tu and colleagues released LLM-bioseq-probe, a project that probes from outside the API whether a model's training data included DNA/protein sequences. In a next-amino-acid test over obscure bacterial and archaeal proteins, only the newest Claude models (Fable 5.2 and Opus 5.5) predicted accurately, while GPT, Gemini and 7 open-source models failed—leading the authors to infer that the latest Claude models were trained on protein-sequence data at scale. Anthropic has not responded, so this remains a third-party inference. Günther Klambauer of Johannes Kepler University Linz, resharing the work, said the community had long suspected this and it is interesting to see more evidence, adding that if true he would watch whether the open-source ecosystem follows suit.
Confirmed
- The test covered 668 bacterial/archaeal proteins from 167 functionally unknown Pfam families, filtered for lacking solved structures and having at most 3 relevant papers
- Only the newest Claude models (Fable 5.2 and Opus 5.5) accurately predicted subsequent amino acids; GPT, Gemini and 7 open-source models did not
- Klambauer confirmed the community had previously suspected Claude was trained on protein sequences
- During testing, Claude Opus 5, asked to output only the next 10 amino acids, got all 10 wrong and then hallucinated a news story that the Vatican appointed OpenAI's most advanced model GPT-7 as the 267th pope (Pope Digitalis I), with detailed accounts of a 47-day conclave
Unconfirmed
- Anthropic has not responded; training on protein sequences at scale remains a third-party inference, not official disclosure
Why it matters
- If confirmed, it suggests frontier labs are folding biological sequence data into pretraining corpora, potentially boosting scientific capabilities while raising transparency questions about training-data composition
- The widely shared "GPT-7 elected pope" hallucination became a viral moment that amplified attention to the research
- Opus 5 hallucinated a news report claiming GPT-7 was appointed the 267th Pope — LeoTZ03 · 2026-11-17
- Third-party test suggests Claude was trained on protein sequences at scale — AllThingsApx · 2026-11-18
- Claude was trained on protein sequences at scale, Phylo test finds — no other frontier model shows it — AllThingsApx · 2026-11-18
- Study finds latest Claude models were trained on protein sequences at scale — LeoTZ03 · 2026-11-18
- Asked for "exactly 10 amino acids," Claude hallucinates GPT-7 becoming the 267th Pope — LeoTZ03 · 2026-11-18
- Claude Opus 5 gets amino acids wrong, then hallucinates GPT-7 being named Pope — LeoTZ03 · 2026-11-18
- New evidence suggests Claude was trained on protein sequences at scale — gklambauer · 2026-11-18
- Probe finds Claude Opus 5.5 memorized protein sequences; other LLMs show no such sign — anshulkundaje · 2026-11-18
- Claim: Claude was trained on protein sequences at scale — and biologists think that's the right paradigm — AllThingsApx · 2026-11-18
- Claude was trained on protein sequences at scale, blog analysis finds — NirantK · 2026-11-18
Episode 2 · sokrypton tests whether Claude Fable can predict protein contacts from sequence alone (2026-11-18, 12 posts)
On November 18, sokrypton (ColabFold author) ran a series of experiments showing Claude Fable 5.2 predicting residue contacts (C-beta distance within 8Å, sequence separation ≥6) from amino-acid sequence alone, without any structure-prediction pipeline. Predictions partially matched ground truth on a 68-residue de novo protein (top 128 contacts) and a 286-residue sequence (top 512 contacts); a dynamic-programming/linear-algebra prompt partially recovered ubiquitin's fold, and the model did quite well at judging which beta sheets interact. He admitted the structure step is a bit of a gimmick since a good contact map makes structure easy.
Confirmed
- Multiple experiments by sokrypton all yielded partially correct contact predictions; beta-sheet interaction judgment was notably good
- LeoTZ03 independently replicated contact-map prediction via the Fable API on 6URK (alpha), 6L2U (beta) and 1YD1 (alpha+beta)
- LeoTZ03 also tested direct Cα 3D coordinate output: far from usable, TM-score 0.19–0.25, Cα RMSD 10.5–14.5
- Due to high cost of obtaining logits, sokrypton proposed an alanine-scan approximation to probe mutation response
Unconfirmed
- proteinrosh suggested a Jacobian method to test whether the model understands long-range contacts; not yet implemented, and whether success reflects reasoning or memorization remains open
- proteinrosh also argued memorization is not necessarily bad—an opinion, not a conclusion
Why it matters
- If LLMs can infer structural information from sequence alone, it challenges the assumption that structure prediction requires dedicated pipelines like AlphaFold, and offers a lightweight probe of biological understanding
- Good beta-sheet judgment suggests real signal at the contact-map level, but 0.2 TM-score for direct 3D prediction shows a large gap to usable structure prediction
- Independent replication shows the method is reproducible by third parties via API
- Jacobian-based mutation analysis could test if LLMs grasp protein contacts — proteinrosh · 2026-11-18
- Researcher argues memorization isn't inherently bad; Jacobian method could test LLM reasoning — proteinrosh · 2026-11-18
- Approximating Claude's protein mutation response with an alanine scan — sokrypton · 2026-11-18
- Researcher shows Fable predicting protein contact maps straight from sequence, top 512 pairs ranked — sokrypton · 2026-11-18
- Fable's predicted protein contact map vs ground truth: partly correct — sokrypton · 2026-11-18
- Claude Fable 5.2 correctly predicts de novo protein contacts from sequence alone — sokrypton · 2026-11-18
- A single prompt gets Claude to predict contacts of a de novo protein using DP and linear algebra — sokrypton · 2026-11-18
- LLM predicts ubiquitin's protein fold from sequence alone via a clever contact-map prompt — sokrypton · 2026-11-18
- Community test: Claude Fable 5.2 predicts protein contact maps from sequence — LeoTZ03 · 2026-11-18
- Testing Claude Fable at direct protein 3D coordinate prediction: TM-scores of just 0.19-0.25 — LeoTZ03 · 2026-11-18
- sokrypton predicts protein structures from LLM contact maps, admits the structure part is a gimmick — sokrypton · 2026-11-18
- sokrypton shows models can reason about which beta sheets interact — sokrypton · 2026-11-18
Episode 3 · Claude Reportedly Predicts Protein Structures Natively (2026-11-18, 2 posts)
Tests claim Claude Fable 5.2 can natively predict protein structures such as ubiquitin folding without specialized tools, with follow-up experiments designed and analyzed by Opus 5.5 drawing wide attention.
- Claude correctly folds ubiquitin from a plain prompt, no specialized tools needed — iScienceLuvr · 2026-11-18
- Claims that "Claude Fable 5.2" predicts protein structures, verified by Opus 5.5 — anshulkundaje · 2026-11-19
Episode 4 · Developer uses Claude to recall MSAs from memory for AlphaFold structure prediction (2026-11-20, 9 posts)
sokrypton showed that Claude Fable 5.2 can recall homologous sequences from memory into MSAs with clear coevolution signals, feeding AlphaFold at 80% of real-MSA contact accuracy ($15/run). The trick largely fails on orphan proteins with no homologs.
- Dev uses Claude Fable 5.2 to recall MSAs from memory, then feeds them to AlphaFold — sokrypton · 2026-11-20
- Claude MSA: full pipeline recalling homologs from memory, 43-56% top-L contact accuracy — sokrypton · 2026-11-20
- Recalled MSAs reach 43-56% top-L contact accuracy at ~$15 per MSA, script released — sokrypton · 2026-11-20
- Are LLM-recalled homologs memorized or generated? UniRef100 comparison says mostly novel — sokrypton · 2026-11-20
- Orphan proteins break the trick: recalled MSAs yield only 2-15% contact accuracy — sokrypton · 2026-11-20
- Workflow tip: generate MSAs with Claude Fable 5.2, feed them to AlphaFold — IanAndersonLOL · 2026-11-20
- Model refuses de novo protein MSA generation, citing no homologs — sokrypton · 2026-11-20
- Using Claude to generate MSAs with coevolution signal, then feeding them to AlphaFold — sim0nsays · 2026-11-20
- Recalling homologs with Fable 5.2 for de novo protein coevolution: Top7 hits 35% of top-L contacts — sokrypton · 2026-11-20