FULL STORY

Carbon-X: Hugging Face's DNA Model Released Amid Doubts

Hugging Face's bio team open-sourced the 14.8B-parameter DNA language model Carbon-X, but its benchmark results were quickly questioned by a Stanford professor, sparking debate over the evaluation's credibility.

2026-10-22 ~ 2026-10-26 · 2 episodes · 36 posts

Episode 1 · Hugging Face Open-Sources DNA Model Carbon-X as Stanford Professor Questions Its Benchmarks (2026-10-22, 33 posts)

On October 22, Hugging Face's bio team (Loubna Ben Allal et al.) open-sourced Carbon-X, a 14.8B-parameter decoder-only DNA language model built on Qwen3-14B that unifies DNA sequences and natural language in a single model, with blog post and weights released and local execution on a laptop possible. Co-founder Thom Wolf said it was trained with TIGEM, the University of Naples Federico II, and Zhongguancun Academy. The team claims it matches GPT-6.1 Sol on five DNA benchmarks at roughly 1/1000 the inference cost. The next day, Stanford professor Anshul Kundaje publicly questioned the rigor of its evaluations, and the BioReason team subsequently admitted answer leakage in the benchmark, which they said has been fixed.

Confirmed

  • Construction: midtraining (continued pretraining) on Qwen3-14B (lvwerra described it as Qwen3.5 14B) with 300B tokens of DNA, English, and interleaved data; a hybrid tokenizer uses BPE for English and 6-mers for DNA in a single token stream. Users can paste raw DNA into a chat and get English answers about sequence function or mutation effects, and the model can generate new DNA sequences for gene-therapy work.
  • Performance: beats all tested open language models across five DNA benchmarks with an average of 66.7 (Thom Wolf rounded to 67%), a 27-point average gain over the Qwen3-14B base, matching GPT-6.1 Sol at 1000x+ lower inference cost (also stated as 1/1800), with mutation effect prediction and DNA sequence completion as strongest tasks.
  • Training methodology: pure RL directly on Qwen3-14B never broke the 50% random baseline on FrontierVEP, while RL from the midtraining checkpoint did, leading the team to conclude midtraining is key.
  • Agent usage: using Carbon-X as a sub-agent for Kimi K3 raised its DNA intelligence score from 49.4% to 60.5%, an 11.1-point gain across five benchmarks.
  • Frontier model refusals: Opus 5.5's next-base prediction score fell from 17.7% at low reasoning to 1.1% at maximum reasoning due to rising refusal rates, with Gemini 4 showing similar trends.

Unconfirmed

  • Evaluation validity is contested: Kundaje called the BioReason variant-effect-prediction evaluations seriously flawed and said other comparisons are mostly meaningless. The BioReason team responded that evaluations covered ClinVar and other existing VEP benchmarks, admitted answer leakage, and said it was fixed with details pending.
  • A circulating independent test claimed Carbon-X underperformed random on 8 of 10 cases, contradicting the team's claims; its setup and provenance are unclear. Whether the "matches GPT-6.1 Sol" claims hold after the leak fix awaits independent verification.

Why it matters

Carbon-X suggests language models can gain large improvements from DNA training at very low cost, validating a midtraining-plus-hybrid-tokenizer route and motivating open alternatives amid frontier models' over-refusal of genomics requests. But Kundaje's critique, the admitted benchmark leakage, and unfavorable independent tests mean its evaluation methodology is not yet accepted by the academic community, and post-fix results should be treated cautiously.

13 more related posts →

Episode 2 · Hugging Face Open-Sources Carbon-X, a 14.8B DNA Language Model (2026-10-25, 3 posts)

Hugging Face Bio and partners released Carbon-X, an open-source 14.8B-parameter DNA language model under Apache 2.0 that lets users query genetic mutations and regulatory functions in natural language, approaching frontier-model performance at a fraction of the inference cost.