UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation
Jiancheng Pan, Yuzheng Liu, Hengyuan Cao, Qiyuan Zhang, Yangyan Li, Biao Gong, Zheng Zhang, Ka Long Cheng, Xinyang Wang, Yujun Shen
cs.CV
2026-11-12
UniSpark uses about 9B parameters per branch and mixes full attention with linear attention, scoring 2470.3 on MME-S, with 1024 image generation a 3.23× speedup over BAGEL.
Understanding reads an image. Generation has to draw one. Interleaved output adds a further constraint: once an image is finished, later text and later edits should condition on it. In a continuous latent space the target is noisy during denoising, and only the finished clean representation belongs in the context that follows. Leave the noise in context, and later predictions see half-finished states. Drop the visual history, and an edit has no reference to stay consistent with.
High resolution and multi-image sequences push the visual token count up. Full attention lets tokens interact pairwise, and the cost grows with the square of the length. Linear attention folds history into a recurrent state and bills length linearly, but it is weaker at fine-grained lookup. A unified model has to run causal text, iterative denoising, and a hard split between reusable clean images and private noisy targets.
UniSpark-9B keeps BAGEL's Mixture-of-Transformer-Experts (MoT): separate parameters by token type, shared attention for context. The 9B figure is the size of each branch. Understanding and generation each hold about 9B parameters. Text and semantic visual features use the understanding expert, with semantic features from the Qwen3.5 vision encoder. Generation targets and clean context use the generation expert. A frozen Wan2.2 VAE encodes them into latents, which are then split into patches.
Layers alternate full attention and GatedDeltaNet. GatedDeltaNet keeps a fixed-size matrix state, updated with a forget gate and a delta rule, so compute grows linearly with sequence length. The two token types use their own projections and share the recurrent state plus a short convolution history. Switching experts does not clear that state.
A block rule keeps noisy targets private. Later blocks can see a clean image and cannot see the noisy block. The noisy block can read its prefix and cannot read its own later clean copy. Full attention enforces this with a visibility mask: causal inside text, bidirectional inside a visual block. Linear attention runs a local update on a noisy block, then discards the state and convolution history. Only text and clean images write back to the public state. Training reads clean images from ground truth. Inference encodes an image only after sampling finishes.
The loss is text cross-entropy plus visual flow matching: interpolate a clean latent with noise and predict the velocity. Four stages run from 256-scale joint modeling to 512, then 1024 through 2K, then supervised fine-tuning (SFT). At high resolution the text loss weight is 0.25, the visual weight stays 1, and interleaved data is turned off. Counted sources are 159.66M generation samples and 213.39M understanding samples (99.98M captions, 97.90M visual QA, 9.99M OCR, 4.51M grounding, 1.00M GUI), plus 97.90M text-only samples, 0.50M edits, and 624.8K interleaved sequences. The understanding total adds five task groups and is not deduplicated by image. A post-SFT video adaptation freezes the understanding branch and samples video generation, video editing, image generation, and image editing at 5:1:1:0.1.
In Table 5, UniSpark posts the highest listed MME-P, MME-S, MMBench, and MMVP scores: 1766.0, 2470.3, 86.7, and 72.0. Against BAGEL (7B MoT) it is higher on all six understanding families, including MM-Vet 73.4 versus 67.2 and MathVista 77.2 versus 73.1.
Qwen3.5-9B, which only does understanding, scores 2439.1 on MME-S, 43.1 on MMMU-Val, and 70.7 on MMVP, but 79.6 on both MM-Vet and MathVista, ahead by 6.2 and 2.4. MetaQuery-XL still leads MMMU at 58.6 versus 56.7. Strong perception does not carry into every reasoning set. MMBench answer extraction also differs: UniSpark uses rules only, while Qwen3.5 and SenseNova-U1 fall back to a Qwen parser.
| Model | Metric | Score |
| UniSpark | MME-S / MMBench / MMMU / MMVP | 2470.3 / 86.7 / 56.7 / 72.0 |
| BAGEL 7B | same | 2388.0 / 85.0 / 55.3 / 69.3 |
| Qwen3.5-9B | same | 2439.1 / 85.1 / 43.1 / 70.7 |
| SenseNova-U1 | same | 2343.5 / 82.8 / 53.1 / 58.7 |
| UniSpark | GenEval / DPG-Bench | 0.83 / 80.54 |
| BAGEL | GenEval / DPG-Bench | 0.82 / 85.07 |
| LLaDA2.0-Uni | GenEval / DPG-Bench | 0.89 / 87.76 |
| Qwen-Image | GenEval / DPG-Bench | 0.87 / 88.32 |
After unified SFT, GenEval overall is 0.83, near BAGEL's direct-generation 0.82 and below LLaDA2.0-Uni at 0.89 and Lumina-DiMOO at 0.88. Single object, color, and two objects score 0.99, 0.94, and 0.93. Counting, position, and attribute binding score 0.68, 0.67, and 0.73. DPG-Bench overall is 80.54, below BAGEL at 85.07 and Qwen-Image at 88.32. Entity is 82.62, attribute 77.07, relation 76.71. Objects and colors hold. Compositional relations do not. Those baselines keep their original eval settings. UniSpark's scores were measured separately.
On all 737 ImgEdit samples, with Qwen3-VL-235B as the judge, the overall score is 3.13. Style is 4.05 and object addition is 3.72. Extraction is 1.68 and hybrid edits are 1.62. The judge differs from the published baselines, so 3.13 is not a rank against Qwen-Image-Edit at 4.35 or BAGEL at 3.20.
On one H20-3e GPU, in BF16 at batch size 1, median generation latency at 1024 drops from BAGEL's 45.80 seconds to 14.16 seconds, a 3.23× speedup. At 512 it drops from 11.76 seconds to 7.27 seconds, a 1.62× speedup. At 2048, BAGEL is unreported. UniSpark takes 51.48 seconds against SenseNova-U1's 58.34, with peak allocated memory of 62.74 GB versus 37.39 GB. At 1024, BAGEL uses 4,096 image tokens and UniSpark uses 1,024, so a higher tokens/s number for BAGEL does not mean a faster image. For understanding, with 128 forced text tokens and 1,024 visual tokens, UniSpark decodes at 22.19 tokens/s and finishes in 6.02 seconds. BAGEL does 36.60 tokens/s and 3.76 seconds. SenseNova-U1 does 18.09 tokens/s and 7.32 seconds. Raising the visual budget from 256 to 1,024 tokens moves UniSpark's total latency only from 5.94 to 6.02 seconds.
Interleaved generation and video have no quantitative scores.
Visual tokens dominate the cost of a unified model. Putting full attention and linear attention in different layers gives pairwise mixing and linear recurrence different jobs. The 3.23× generation speedup versus BAGEL at 1024 is the system number that is easiest to check. At 2K the cost returns as memory: 62.74 GB peak allocated.
Understanding survives joint training. MME-S sits above the listed unified models and slightly above Qwen3.5-9B. Generation is a small step. GenEval matches BAGEL, and DPG-Bench still trails dedicated text-to-image models. One set of weights can answer questions, synthesize at 2K, and follow edit instructions. It does not replace the strongest specialized generator.
Scale, training data, and eval protocols all differ, so the understanding gains cannot be assigned to hybrid attention alone. The latency tables do not isolate the attention design either. SenseNova-U1's 512 and 1024 timings sit outside its recommended training resolution buckets.
ImgEdit uses a different judge. The 3.13 score only sorts UniSpark's own categories: style is decent, extraction and hybrid edits are weak. The interleaved figures were chosen from 12 paired runs, and flame direction plus jellyfish contraction are still off. Video adaptation freezes the understanding branch and reports no metrics.
The 213.39M understanding count sums task groups, so overlap is uncounted. High-resolution training drops interleaved data, and the SFT hyperparameters are marked preliminary. There is no ablation on how full attention and GatedDeltaNet are layered, or on how much discarding the noisy state matters. The related work already separates joint task support from cross-task gain. UniSpark does not test that gain.