CoBit paper: one continuous bitstream diffusion recipe spans language, image, speech and protein generation
LucaAmb · x · 2026-11-01
The paper Multimodal Co-Generation With Continuous Bitstream Diffusion (arXiv:2610.14349; authors include Mark Girolami and Luca Ambrogioni) shows that CoBit, originally built for language modeling, provides a unified framework for joint multimodal generation: encode each modality as bits, flatten, denoise—almost nothing has to change.
Each domain is trained from scratch but shares one recipe (single input projection, transformer backbone, bitwise prediction head, binary score-matching objective); a single checkpoint supports joint and conditional generation:
- Image-text: near-perfect conditional accuracy on MNIST-Sum, 99.46% mutually consistent joint samples, no learned tokenizers
- Speech-text: 633M params, no text pretraining; one model does synthesis, recognition, continuation and co-generation with 3.5% WER between generated text and transcriptions of generated speech
- Protein sequence-structure: one checkpoint handles forward folding, inverse folding and joint generation, 0.911 mean scTM for generated pairs
Takeaway: surprisingly little needs to change across modalities—bitstream representations are a viable route to unified co-generation.
More from Multimodal
- Halloween meme: AI-generated pumpkins turn and smile wider than you carved them — LudovicCreator · 2026-11-01
- VFX Supervisor Open-Sources 20 Skills for Building an Automated Motion Design Pipeline with Claude Opus 5.5 — GCWebDesigner · 2026-11-01
- Open-H3-Regenerate-2K tested: sharper faces but up to 2x generation time — gutster_95 · 2026-11-01
- AI-made Halloween special drops with action, jokes and horror easter eggs in 4K — charis_ai · 2026-11-01
- AI filmmakers are making movies Hollywood never would, on minimal budgets — DavidmComfort · 2026-11-01
- IronLeo78 MTV: a two-node ComfyUI pack that turns any song into a music video — IronLeo78x · 2026-11-01