Cherimoya Genome Model Hits 0.78 on Counting, but Cross-Species Gains Fail to Transfer

The Cherimoya genome model team recently shared results and limitations from their new tuned version: counting performance improved from about 0.72 to roughly 0.78, and variant effect prediction (VEP) error was nearly halved; however, the authors acknowledge that gains from multi-species track prediction did not transfer to the variant level or to E2G-style benchmarks, where performance is only on par with Borzoi. This mix of progress and limitation highlights the central debate over generalization in current genomic language models and deserves attention.

Confirmed

• Team member Joachim Schreiber disclosed that, through dual optimization of architecture and data, Cherimoya's counting performance rose from about 0.72 to about 0.78, with VEP error nearly halved.
• Specific changes include parameter tweaks such as reducing max jitter from 500 to 50.
• Author Bruno de Almeida, responding to feedback from Stanford's Anshul Kundaje, noted that the new version added GTEx eQTL (Figure 5e) and E2G (Supplementary Figure 21) evaluations, but the gains from multi-species track prediction did not carry over — performance on these benchmarks only matches Borzoi.
• Joachim Schreiber added his experience: in many cases, improvements on observational holdouts correlate poorly with variant effect prediction performance, even for short-range models; he suspects this is related to the homolog leakage issue raised by Muntakim Rafi et al.

Why it matters

• The marked improvements in counting performance and VEP error show that tuning architecture and data mixtures still pays off, and details like the max jitter change may be useful references for the community.
• The disconnect between observational benchmark performance and variant prediction, along with cross-species gains failing to transfer, is a key methodological concern for evaluating current genomic models (especially post-Borzoi work): improvements on standard holdouts do not necessarily mean better functional variant interpretation, and homolog leakage may be one reason.

2026-10-30 ~ 2026-10-31 · 5 related posts

• [source] Genomic model authors: multispecies gains don't transfer to variant effect tasks — deAlmeida_BP · 2026-10-30
• Observational benchmark gains poorly correlate with variant effect prediction — jmschreiber91 · 2026-10-30
• Cherimoya's multispecies gains fail to transfer, only matching Borzoi on eQTL and E2G evals — EthanArmand2 · 2026-10-31
• [source] Cherimoya genomics model lifts count score to 0.78, nearly halves VEP error — jmschreiber91 · 2026-10-31
• Tuning the Cherimoya genomics model: count score rose from 0.72 to 0.78 — jmschreiber91 · 2026-10-31