HuggingFace Open-Sources FineDNA Dataset and Carbon Models, Beating GPT-6.1 on DNA Benchmarks

Hugging Face's biology team (Loubna Ben Allal et al.) released FineDNA, a genomic dataset, along with the open Carbon model family, publishing all data, model weights, and training recipes (Apache 2.0). Trained on them, Carbon-X surpasses GPT-6.1 and all open-source language models on DNA benchmarks, making it the strongest open model for DNA tasks.

Confirmed

• The FineDNA dataset spans 300B+ tokens, mixing genomic sequences, papers, and web text seeded with DNA, plus 200k SFT samples covering DNA tasks such as variant effect prediction and sequence QA
• Carbon-X was built from Qwen3-14B via FineDNA midtraining + 200k SFT + RL with verifiable rewards; the authors report it outperforms all open-source language models and GPT-6.1 on DNA tasks
• Key midtraining recipe points: the authors emphasize that with a suitable learning rate, DNA scores keep climbing during midtraining (0.56 sequence recovery, 0.92 ClinVar variant effect AUROC) while English capability stays at its starting level with no forgetting
• The authors also note that midtraining is a prerequisite for RL to work: RL applied directly to Qwen3-14B leaves FrontierVEP stuck at random-chance level (50%); only after FineDNA midtraining does the same RL recipe deliver sustained gains
• The main Carbon family includes Carbon-500M/3B/8B base models, usable zero-shot and fully open-source as well

Why it matters

• With data, weights, and the full training recipe all public, this provides a reproducible open-source path for genomics
• The empirical findings that midtraining is a prerequisite for effective RL, and that English capability is retained without forgetting, offer methodological value for midtraining + RL pipelines in other domains

2026-11-01 ~ 2026-11-01 · 6 related posts

Full story(3 episodes)→

• Episode 1: Hugging Face Open-Sources DNA Model Carbon-X as Stanford Professor Questions Its Benchmarks(2026-10-22, 33 posts)
• Episode 2: Hugging Face Open-Sources Carbon-X, a 14.8B DNA Language Model(2026-10-25, 3 posts)
• Episode 3: HuggingFace Open-Sources FineDNA Dataset and Carbon Models, Beating GPT-6.1 on DNA Benchmarks(2026-11-01, 6 posts)

• Hugging Face releases FineDNA: 300B+ token dataset to teach LLMs DNA — LoubnaBenAllal1 · 2026-11-01
• FineDNA midtraining lifts DNA scores without degrading English — LoubnaBenAllal1 · 2026-11-01
• RL fails without midtraining: Qwen3-14B stays at chance on FrontierVEP — LoubnaBenAllal1 · 2026-11-01
• [source] Carbon-X: midtrained Qwen3-14B beats every open LLM on DNA benchmarks — LoubnaBenAllal1 · 2026-11-01
• [source] Hugging Face open-sources Carbon DNA foundation models with full training recipes — LoubnaBenAllal1 · 2026-11-01
• [source] HuggingFace releases FineDNA dataset, turning Qwen3-14B into a DNA model that beats GPT-6.1 — loubnabnl · 2026-11-01