HuggingFace opens its FineDNA recipe: mid-training on 300B tokens breaks the distillation ceiling
_lewtun · x · 2026-11-03
Lewis Tunstall of HuggingFace announces that the team's full mid-training and post-training recipe for FineDNA is now open, hoping the lessons on breaking past the distillation ceiling help model builders do the same in new domains.
A key caveat: none of this works unless the model already knows something about the domain. Off the shelf, Qwen3-14B gave a learning signal on only 17% of prompts; after mid-training on 300B tokens of DNA and text, that rose to 42%. The dataset is publicly released.
Related event: HuggingFace Open-Sources FineDNA Recipe, Bypassing Teacher Distillation(4 posts)→
More from Research
- After two silent grant rejections, researcher posts federated learning paper on arXiv — wandedob · 2026-11-03
- Scale AI's READY framework: identical-accuracy agents need 39.2% vs 29.6% human review — demian_ai · 2026-11-03
- LLMs can natively edit neural network weights: 0.016% of changes boost small model math 6x — Dr_Singularity · 2026-11-03
- Developer warns: do NOT use weight-space programming — djcows · 2026-11-03
- Lean 4 proof shows every finite simple group is generated by two elements — geoffreyirving · 2026-11-03
- Musk: diffusion wins on any bitstream; CoBit applies one diffusion recipe to text, speech, images, proteins — LucaAmb · 2026-11-03