WeLM-Image: A Unified Image Generation Model towards Visual Intelligence
Canmiao Fu, Chen Li, Guohui Zhang, Hao Lu, Hongyang Wei, Huadong Wu, Huaisong Zhang, Hubery Yin, Liyuan Cui, Qixin Yan, Tong Shao, Xiaoqiang Zhou, Xinhang Leng, Ying Zhang, Zhenhai Liu, Zhipeng Huang
cs.CV, cs.AI
2026-11-05
WeLM-Image unifies T2I, editing, and multi-reference: 31.7B stored, 5.4B active. It leads reported open-source on three benches; eight-step Turbo draws 1024px in 4.98s on H20.
Consumer image apps get stuck on control, long after raw image quality stopped being the bottleneck. One model has to follow long Chinese and English constraints, render the words inside the picture correctly, edit from a single instruction without losing the subject, and separate identity, clothing, and scene across several references. Flow and diffusion models already lifted prompt alignment. What is left is that control, at a cost one GPU can actually serve.
Qwen-Image conditions synthesis with a multimodal language model and keeps reconstructive features so edits stay consistent. HunyuanImage 3.0 puts language modeling and image generation in one shared backbone. SenseNova-U1 learns from pixels and words together. Next-token text and continuous images pull optimization in different directions, and fully shared weights make them interfere. WeChat also caps scale by single-GPU deployment and per-request cost. WeLM-Image puts text-to-image, instruction editing, and multi-reference synthesis on one interleaved sequence.
Each sample is text and image segments in source order. Zero references means text-to-image. One or more means editing or multi-reference composition. Understanding samples are an optional image plus a question, with cross-entropy only on the answer. Samples pack to 64K tokens and cannot attend across the pack boundary.
Every image goes through two encoders. A ViT on the understanding path reads objects, relations, and layout. WeLM-Image-VAE on the generation path stores a latent that can be reconstructed. References keep both views. The target is only a noisy latent, and flow matching predicts the velocity from noise toward the clean latent. Condition tokens are computed once and cached. Each denoising step recomputes only the noisy target, so extra references do not multiply the cost of every step.
The backbone is MoT–MoE: a Mixture-of-Transformers with its own Mixture-of-Experts on each path. Understanding and generation keep separate projections, normalization, and experts, but every layer runs one shared attention block, so a target query still reads the text and the references. Separate weights keep the two losses from dragging on each other. Separate expert pools stop the router from mixing them back together. There are 27 layers and a hidden size of 2,048. Stored size is 31.7B including the ViT and excluding the VAE. An understanding token selects 6 of 64 routed experts and activates about 2.2B parameters. Each target latent token selects 20 and activates about 5.4B. Text and ViT tokens take the understanding path. Clean references and the noisy target take the generation path. Keys and values of the noisy target stay hidden from every other segment, so the answer cannot leak into the condition. Text uses coordinate (p, p, p), and the three-axis rotary embedding then matches the pretrained one-dimensional RoPE. Only VAE cells add a centered height and width around that anchor.
VAE training has three stages: reconstruction, alignment of semantic geometry to DINOv2, then decoder-only tuning on text-heavy images. Both paths initialize from Kimi-VL. Cross-entropy and flow matching are weighted 0.25 to 1. The flow loss is averaged over each whole image, so a 1024 target and a 1536 target contribute the same amount.
The raw pool is about 5 billion images, filtered to about 0.6 billion. Of the kept images, 59.74% fall in the 1024-or-higher resolution groups. A knowledge-graph vocabulary samples named concepts by quota so web frequency does not drown the long tail. Clustering covers appearances that still have no stable name. Editing assigns a visual cluster to a task before the instruction is written, and keeps a real photograph on the target side when possible. Multi-reference examples use 2 to 4 images across 18 role tasks, drawn from an entity pool of about 750,000 images. Training moves from 512 to 1024 to 1536, then supervised fine-tuning. The first two stages still include 20% understanding data. After that the share is zero. Fine-tuning is 60% text-to-image, 30% editing, and 10% multi-reference.
Post-training splits text-to-image and editing into two reinforcement-learning runs. The backbone stays frozen. Updates touch generation-side LoRA only, and ablations show that extending those adapters into the MoE layers helps both tasks. Text-to-image uses four rewards. The alignment reward is a rubric written before rollout, without looking at the image. Editing uses a checklist. The score is prompt-following times (0.4 + 0.4 times consistency + 0.2 times visual quality), so a missed instruction cannot be rescued by a clean picture. Twenty-seven edit types are sampled easy, medium, and hard at 3:5:2. Noise-aware on-policy distillation then merges the two specialists: a prompt-enhanced teacher supervises layout at high noise, and the standard RL teacher supervises local detail at low noise. MeanFlow cuts the sampler to eight steps and drops guidance. That checkpoint is WeLM-Image-Turbo.
In the reported overall comparisons, the 1536-resolution model ranks first among the open-source models evaluated on PRISM-Bench, Qwen-Image-Bench, and UnicBench. The introduction also places it first on UnicBench and WeGenEditBench among those open-source models. The abstract states the rank and does not state the margins.
| Setup | Metric | Result |
| Turbo, 8 steps, no guidance, one H20 | 1024 image | 4.98 s |
| Versus the 50-step CFG base | Speedup | 11.4× |
| Turbo editing, 1–4 inputs | 1024 output | 6.78–13.17 s |
| Full model / generation path | Stored / active per latent token | 31.7B / about 5.4B |
| Understanding path | Active per token | about 2.2B |
WeLM-Image-VAE has 83.75M parameters, 8× spatial compression, and 16 channels. It has the highest reconstruction PSNR of Table 1 on all three sets: 34.89 on COCO at 512, against 34.20 for FLUX-VAE and 33.60 for Qwen-VAE; 34.47 on in-house posters at 1024, against 33.91 for FLUX-VAE; 38.49 on ImageNet at 512, against 37.99 for FLUX-VAE. SSIM is not first. FLUX.2-VAE, with 32 channels, reaches 0.9540 on COCO, where this VAE reaches 0.9498.
Each VAE is then frozen and a LightningDiT-XL/2 is trained for 80 epochs on ImageNet at 256, with 50k samples and no guidance. Among 16-channel tokenizers, WeLM-Image-VAE is best: FID 12.21 and IS 97.96, against Qwen-VAE at FID 15.29 and IS 84.62, and FLUX-VAE at FID 27.63 and IS 55.10. The 32-channel FLUX.2-VAE reaches a lower FID, 9.87. Hunyuan-3.0-VAE, about 1.26B parameters, is left out of that generation run because the autoencoder is larger than the diffusion head. Its reconstruction PSNR is also below the 8× models.
Single-GPU serving is the constraint the architecture is built around. Caching the condition, activating about 5.4B parameters on the generation path, and sampling in eight steps without a guidance branch all avoid recomputing a long multi-reference context at every denoising step. Figure 1 plots throughput against the number of input images because that curve, not a single text-to-image latency, is the serving question.
Dual-path transformers, expert mixtures, flow matching, rubric rewards, and few-step distillation each have precedents. The addition here is a pipeline that runs on one H20, with data and rewards aimed at web-frequency bias: concept quotas, reverse-built edit pairs, and a detector for Chinese glyph errors. The editing gate is written into the formula. Visual quality cannot pay for an instruction the model did not follow. A 1024 image in 4.98 seconds fits an async consumer queue. It does not fit a chat box that must answer as the user types. Editing grows from 6.78 seconds to 13.17 seconds as the input count goes from one to four.
At 1536 pretraining and in supervised fine-tuning, the understanding share in the mixture is zero. The report says this version still centers on generation and editing, and leaves stronger visual understanding for a later version. At high resolution the shared backbone is a generator that borrows an understanding path.
The rewards are VLM judgments. The authors say general VLMs miss the structure of Chinese glyphs, so they train a separate stroke detector for that one axis. They do not report how the other rewards agree with human preference. A biased judge is written into the policy.
First place holds only inside the open-source set they evaluated. The abstract gives no score gaps, and closed models are not in that sentence. WeGenEditBench cannot be recomputed elsewhere unless the same items are public. The 11.4× figure is against their own 50-step guided base, not against other few-step samplers.
The concept vocabulary, the aesthetic scorer, and the 0.6-billion-image pool sit inside WeChat. One multi-reference path synthesizes the target with another editor, so that teacher shapes the target distribution. Generation and editing are trained apart because mixed training does not reach either ceiling. After the two policies are merged, the report's method section does not say how much each specialist loses.