How Carbon-X was built: hybrid BPE + 6-mer tokenizer over 300B DNA-text tokens
LoubnaBenAllal1 · x · 2026-10-22
Carbon-X was built by first midtraining Qwen3-14B on 300B tokens of DNA, English and interleaved data. A hybrid tokenizer handles both modalities: BPE for English and a 6-mer tokenizer for DNA sequences.
More from Research
- IBM's LASAGNA study shows RAG technique gains shrink when combined: +47% predicted becomes +17% realized — ibm · 2026-10-22
- Dev hot take: mechanistic interpretability is 'astrology for AI people' — dejavucoder · 2026-10-22
- eLife 'Landmark' paper: RNNs reveal a mechanistic theory of planning in prefrontal cortex — ShahabBakht · 2026-10-22
- Mathematician Kontorovich: 'All code needs to be Lean formalized like yesterday!' — ludwigABAP · 2026-10-22
- Cohere's CulturalRiddles: 28 LLMs score 79.3% on cross-cultural riddles, but just 22% in some cultures — Cohere_Labs · 2026-10-22
- Berkeley-Stanford preprint: agents that periodically forget beat ones that remember everything — ombre-ai · 2026-10-22