No token prediction, no image quantization: MF-1 models text and images as one continuous flow

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces

Hongyuan Tao, Xinggang Wang, Lianghui Zhu, Yongkang Li, Yunchao Wei, Bin Feng, Shaoyu Chen, Qian Zhang, Chang Huang, Kai Yu

cs.CV

2026-10-01

Text and images modeled as continuous chunks under one Flow Matching objective; a 1.6B model with 150B tokens hits 0.82 GenEval and beats discrete and hybrid rivals at matched budgets.

What problem this solves

Unified multimodal models want one network for both understanding and generation, and current designs split into two camps. Fully discrete models (Chameleon, Emu3, Show-o) quantize images into visual tokens and share one categorical sequence with text; training is uniform, but visual fidelity is capped by the tokenizer, and whatever quantization discards is gone for good. Hybrid models (Transfusion, JanusFlow) keep autoregressive text prediction and run diffusion or flow for images; vision stays continuous, but the system carries two objectives, two perturbation schedules, and two samplers.

The third route is fully continuous: keep text and images in embedding space and share one generative objective and sampling procedure. The ingredients existed. ELF showed text embeddings can be modeled directly with Flow Matching, and representation autoencoders supply semantic visual latents. What was missing was wiring both into a randomly initialized backbone for mixed multimodal pretraining. That is the gap Multimodal Flow fills; it comes from HUST's Vision Lab with Beijing Jiaotong University and Horizon Robotics, and the code and weights are public.

Method

Both modalities are carved into hyperchunks and modeled by one chunk-causal flow backbone:

• Continuous representations: a frozen T5-small encoder maps text in 8-token blocks to 512-dim vectors, preserving block order; a frozen SigLIP2 maps each 224×224 image to a 16×16 grid of 256 vectors, preserving spatial layout. One image is a visual chunk, one text block a text chunk.
• Chunk-causal factorization: the sequence distribution factors in chunk order. Predicting chunk k sees only the preceding clean chunks; positions inside the target chunk are modeled jointly. Attention projections are shared across modalities for interaction, while FFNs stay modality-specific, and the ablation shows sharing FFNs is what hurts, especially image-conditioned text.
• Parallel training: each target chunk gets independent noise and its own flow timestep, and the loss is velocity MSE on the regressed clean endpoint. The causal mask lets many target chunks be predicted in one forward pass, with sequence packing for efficiency.
• Sequential inference: the next chunk is integrated from Gaussian noise to data conditioned on the prefix, then appended to the context; KV of finished chunks can be cached. VQA runs 16 sampling steps per text chunk, text-to-image 64.
• Frozen decoders: a separately trained 6-layer bidirectional Transformer maps text latents back to tokens block by block, and a pretrained representation decoder reconstructs images.

Switching tasks changes only chunk content and ordering, never the objective: text-only, image-to-text, text-to-image, and VQA all train with the same Flow Matching loss, finetuning included. CFG works in both directions, peaking at guidance scale 5 for text-to-image and 3 for captioning.

Results

MF-1 is a 1.6B backbone pretrained from scratch on 150B tokens (70% text-only, 20% image understanding, 9% text-to-image, 1% image-only) and finetuned on 5B.

Text-to-image (GenEval overall):

MF-1 adds 83.44 on DPG-Bench, averaging 82.8 across the two benchmarks, and the paper reports the strongest compositional generation and long-prompt alignment among unified models.

Against other train-from-scratch unified models the understanding gap is wide: MMBench 67.2 vs Muddit's 28.4, SEED-Bench 62.4 vs 13.0, VQAv2 72.6 vs D-DiT's 59.5, while those two consumed roughly 400B and 4.3T tokens against MF-1's 150B. LLM-initialized models still lead: Janus-Pro (1.5B, 2T tokens) beats MF-1 on VQAv2 79.8 to 72.6 and MMBench 74.9 to 67.2. The VQAv2/MMBench/POPE average is 75.3.

The controlled comparison is the strongest evidence: same data, optimization schedule, and trainable-parameter budget (1.6B, 50B pretraining plus 5B finetuning).

Transfer from pretraining is real: against a randomly initialized model trained directly on 155B downstream tokens, 150B of pretraining plus 5B of finetuning lifts GenEval from 0.527 to 0.821 and SEED-Bench from 31.6 to 62.4.

One ablation matters beyond this paper: at this budget, semantic representations (DINOv2, SigLIP2) beat reconstruction-oriented VAE latents (FLUX.2, SD-VAE) as the generative state for both generation and captioning, with raw pixels worst. DINOv2 tops generation; SigLIP2 balances generation and captioning.

Why it matters

This moves fully continuous unified modeling from concept to controlled evidence: it beats Transfusion-style hybrids and Chameleon-style discrete models across the board at matched budgets, so the single continuous objective gives up nothing on either side. For practitioners it means a simpler pipeline, with no tokenizer to tune and one sampler to maintain, at 150B tokens, far below what industrial unified models consume. The semantic-versus-VAE-latent ablation stands on its own for anyone building text-to-image systems.

The honest positioning: understanding scores still trail LLM-initialized Janus models. This paper establishes the paradigm as viable and competitive, not superior.

Limitations

The authors defer interleaved long sequences, video, and further modalities to future work. Beyond that:

• Images are fixed at 224×224 with a 16×16 grid, a low ceiling for generation resolution, and the text encoder is the small T5-small.
• The quantization bottleneck is traded for a frozen-encoder bottleneck: high-frequency detail SigLIP discards is equally unrecoverable, and the paper never confronts this analogy directly.
• Inference cost is undisclosed. Text arrives 8 tokens per chunk with 16 sampling steps each, so long answers may be clearly slower than autoregressive decoding; no latency numbers appear.
• Language quality is measured only by GPT-2-large perplexity, with no autoregressive baseline on text benchmarks, and long-range coherence of continuous text generation is untested.
• The gap to Janus-Pro on OK-VQA (39.1 vs 48.7) may mostly reflect data volume (150B vs 2T tokens), but no scaling extrapolation is offered.

Terms

• Flow Matching: training continuous generative models by learning a velocity field from noise to data; a diffusion relative with straighter sampling paths.
• hyperchunk: the paper's generative unit, an 8-token text block or one image's 256 patch vectors; positions within a chunk are joint, chunks generate in order.
• quantization bottleneck: detail lost when compressing images into a discrete codebook can never be recovered downstream.
• Transfusion: the representative hybrid, next-token cross-entropy for text plus diffusion for images in one transformer.
• classifier-free guidance (CFG): extrapolating between conditional and unconditional predictions at inference to strengthen condition adherence; one formula serves text and image generation here.
• representation autoencoder: using a pretrained visual encoder rather than a VAE to provide the continuous generative space, keeping semantics and spatial structure.

Source

• arXiv 2609.40362v1
• PDF
• Hugging Face
• GitHub · https://github.com/hustvl/Multimodal-Flow

Related papers

• Discrete codes first: DeepMind scales autoregressive image models to coherent 256×256 generation
• BrickWright: a LEGO SDK and validator take agents from 9–69% to 100% physical validity
• Best of 22 VLMs hits 58.5% on AgentVQA as visual-state errors dominate
• LoRA on 33B MiniMax-H3 cuts missed entity-count changes from 34.7% to 21.3%
• Qwen3-VL-8B training lifts RewardData artifact Finding F1 from 3.38 to 35.42
• RTAformer gains 4.8 SGDET R@10 on transition frames over matched OED

All paper explainers