Transformers Discover Molecular Structure Without Graph Priors
Tobias Kreiman, Yutong Bai, Fadi Atieh, Elizabeth Weaver, Eric Qu, Aditi S. Krishnapriyan
cs.LG, cond-mat.mtrl-sci, physics.chem-ph, q-bio.BM
2025-10-03
A plain Transformer with no graph or symmetry priors, trained on OMol25, learns electrostatic attention decay and matches the equivariant GNN eSEN at equal training compute.
Machine learning for atomistic modeling has spent a decade converging on graph neural networks: atoms are linked into a graph, usually by connecting pairs within a hand-picked radius cutoff of 6-12 Å, and priors like rotation equivariance are built into the architecture. This is data-efficient, but every new chemistry often demands new hand-designed biases, and a fixed cutoff that suits a dense molecule is wrong for a sparse one. The prior becomes the ceiling.
In parallel, the Transformer unified language, vision, and robotics with no domain priors, and scaling laws made its gains predictable. Could it do molecules the same way? Before datasets like OMol25, the default answer in the field was no, that models without geometric guidance cannot recover complex interactions. The UC Berkeley team ran the experiment and got the opposite answer.
Learned physics. In early layers, attention decays with distance as roughly 1/r², the scaling of electrostatic force; deeper layers flatten to roughly 1/r, the scaling of electrostatic energy. The local focus falls off in the 6-12 Å range, right where graph models put their hand-picked cutoffs. The effective radius adapts: more isolated atoms get larger radii, and later layers track the electrostatic reference, which a fixed-cutoff GNN cannot do. UMAP projections of element embeddings recover periodic-table structure. In a synthetic setup with energies drawn from 1/r^α potentials (α ∈ {1,2,4}), the learned attention exponent correlates with the true exponent at R² = 0.98 and converges to it with more data and model size, using about 1/1000 of the full OMol25 data.
Accuracy. Following the scaling laws to a 1B-parameter model, trained with the same FLOPs budget as the 6M-parameter equivariant GNN eSEN on the same OMol25 data, the Transformer matches eSEN's energy and force MAE on out-of-distribution compositions (eSEN energy MAE: 117.99-129.77 meV) while training and running faster in wall-clock time. Molecular dynamics is cleaner: across 10 validation molecules simulated at 500K for 100 ps, its interatomic-distance distribution h(r) has an MAE of 0.040 against the UMA-S reference, better than either eSEN variant, and a fine-tuned conservative version conserves energy in NVE runs.
| Model | h(r) MAE (vs UMA-S) |
| eSEN-sm-d | 0.077 |
| eSEN-sm-c | 0.065 |
| Transformer (1B, no priors) | 0.040 |
Symmetry and scaling. With no equivariance built in, force predictions under rotation reach cosine similarity above 0.99. Even with positional embeddings injected to break permutation symmetry, reordering atoms changes energy MAE by under 5 meV, less than 5% of the model's error. Scaling laws fit on 5M, 30M, and 90M models predict the 306M model accurately; IsoFLOP curves are parabolic; pre-training scaling shows no saturation at 1B parameters (the 1B run took roughly 750 A100 hours). The pretrained model can also score new structures by log probability, and those scores correlate with the force errors of MACE-OFF, GemNet, and EScAIP on SPICE, a bonus uncertainty estimator.
The paper is an existence proof: given enough data, many physical inductive biases that were engineered by hand can be learned, in forms more flexible than hard-coding (adaptive radii, a kNN attention head, heads that attend more to distant atoms). For practitioners it offers a parallel track to GNNs: a standard architecture inherits FlashAttention, compilers, and GPU tooling, so a 1B dense model beats a 6M sparse GNN in wall-clock time, and scaling laws make returns predictable. This is the most systematic test of Sutton's bitter lesson in atomistic modeling so far.
The accuracy story is parity, not superiority: it matches eSEN at equal compute on one dataset and does not set a new SOTA.
The authors' own list: fine-tuning conservative forces (forces as energy gradients) made training noticeably unstable; more rigorous evaluation is needed before large-scale scientific use; spin tokens get almost no attention because high-spin molecules are rare in the data; the appendix's 1/r^α construction is intuition, not proof.
Two more concerns from the reading. Reading attention scores as interaction strength is indirect evidence; softmax normalization and the distance distribution of atom pairs both contaminate the quantity, and the authors themselves attribute the flattening beyond 20 Å to sparse pairs. The parity claim rests on matched compute and a single dataset, while the core selling point of equivariant GNNs is small-sample efficiency, and the 1B-versus-6M comparison shows exactly what priors buy in data savings. Scaling is verified only to 1B; extrapolating to hundreds of billions of parameters is speculation, and power laws have broken in other modalities before.