RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling
Ziyuan Wang, Bohao Tang, Fei Zhang, Shuo Han, Pengfei Liu
cs.LG, q-bio.GN
2026-08-24
RiboSpan is a 1.61B RNA encoder pretrained natively at 10,240 nt. Global recovery is 94.8% at 10K context; frozen 10-NN on long RNAs is 89.1% vs 81.0% for RNA-FM.
Mature mRNAs often run longer than 1,000 nt, and the 5′ UTR, CDS, and 3′ UTR are coupled: a change in one region can alter folding, stability, and translation that look like they belong to another. Bidirectional RNA encoders such as RNA-FM, RiNALMo, and AIDO.RNA are typically pretrained at about 1,024 tokens, so longer transcripts are truncated. Linear-complexity hybrids (HydraRNA, RNAret) and BPE compression (BiRNA-BERT) stretch coverage by giving up either dense all-to-all nucleotide attention in every layer or fixed single-base resolution. Causal decoders such as EVA can generate at transcript scale, but each position sees only the upstream sequence.
The missing combination is specific: single-nucleotide tokens, dense bidirectional attention in every layer, billion-scale capacity, and native pretraining at full-transcript length. Position interpolation and YaRN can stretch RoPE at inference. They do not expose the backbone to transcript-scale interactions during pretraining.
RiboSpan is a 32-layer pre-norm Transformer encoder: hidden size 2,048, 32 heads, SwiGLU, RoPE over the full head dimension, 1.61B parameters. One nucleotide is one token; the vocabulary has 16 entries. The corpus mixes RNAcentral v26.0 with Ensembl 115 / Ensembl Genomes 62. After filtering and exact deduplication it holds 67.6 million sequences and 85.7 billion nucleotide tokens. Training uses Megatron-LM with variable-length packing: several RNAs share a window, attention is cut at sequence boundaries, and positional indices reset, so packed sequences do not leak into each other.
The controls are tight. The same data and architecture are trained at 1,024 and 10,240 tokens. A 15% masking stage runs for 6 epochs, then a 40% masking continuation runs for 2 epochs from that checkpoint. The heavier mask is there to support later discrete diffusion.
On the same backbone, a conditional masked-diffusion module (MDLM-style, AdaLN-Zero blocks) jointly rewrites the 5′ UTR, CDS, and 3′ UTR. CDS edits are restricted to synonymous codons, so the protein sequence stays fixed. Property predictors and RL post-training are described as a forthcoming journal paper; this preprint has no generation numbers.
Reconstruction splits context length from mask rate. At 10,240 tokens and 15% masking, native 10K-15 recovers 94.8% of nucleotides globally; a 1K-15 model extrapolated to the same length gets 93.1%, AIDO.RNA-CDS 91.6%. Under 40% masking the gap widens: 10K-40 reaches 85.9%, 1K-15 extrapolation 77.2%. MLM loss agrees: 0.724 for 10K-15 at 10,240 / 15%, versus 1.151 for AIDO and 0.982 for 1K-15 extrapolation.
The long-context benchmark is a controlled probe. Complete mRNAs are grouped at 1,024 through 10,240 nt, 10 transcripts per bin. Only a centered interval of width L/32 is rearranged, composition-preserving, with the rest of the sequence untouched. Three quantities are tracked: extra regional separation after the edit (ΔCS), same-base similarity across the cut (Ccross), and how much unchanged distal positions move (Ddistal).
| Model | Setting | ΔCS↑ | Ccross↓ | Ddistal |
| RiboSpan-10K-15 | Native 10K | 0.406 | 0.303 | 0.0008 |
| HydraRNA | Direct | 0.314 | 0.522 | 0.0007 |
| AIDO.RNA-CDS | YaRN | 0.446 | 0.317 | 0.025 |
| RiboSpan-1K-15 | YaRN | 0.451 | 0.334 | 0.016 |
YaRN restores regional separation for short-context dense models, sometimes above native 10K, but distal diffusion rises by an order of magnitude. HydraRNA barely moves the far positions and also separates regions more weakly. Native 10K holds both sides.
Frozen 10-NN, no trained head: on 89,955 held-out sequences across 25 biotypes, 10K-15 reaches 89.9% accuracy against 86.5% for RNA-FM and 86.1% for HydraRNA. On RNAs longer than 1,024 nt, 10K-40 hits 89.1% versus 81.0% for RNA-FM. On the 20 most frequent Rfam families, RNA-FM still leads (99.0% vs 98.4%).
Anyone scoring or designing full-length transcripts is still mostly chopping them into 1K windows. This paper actually trains the expensive option: native long context with single-nucleotide dense attention. The diagnostic is the part to steal. Extrapolation and YaRN can repair some numbers while spilling interaction into distant, unedited bases. Frozen evaluation also shows the backbone already organizes RNA types, so a task head is not a prerequisite for measuring representation quality. The generation stack is a design sketch, not a tool yet.
The long-context panel has 10 transcripts per length and 50 in total; the statistics are thin. Frozen classification uses the authors' own pretraining hold-out, not an independent public set, so comparisons with RNA-FM partly reflect corpus overlap. Generation, property prediction, and RL post-training have no numbers here. Train and inference cost for dense 10K attention is almost unreported. HydraRNA uses attention in only 2 of 12 layers, so it is a limited stand-in for the whole efficient-architecture family.