GAS from ByteDance: use generation purely as training-time supervision, discard the branch at inference for zero overhead

Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

Zhongbin Guo, Jiahao Xie, Dongling Xiao, Qianle Wang, Ruiqi Lu, Xiaomin He, Wanxuan Sun, Cheng Yang

cs.CV

2026-08-13

A ByteDance team reframes visual generation as auxiliary supervision: a decoupled Mixture-of-Transformers branch predicts continuous visual embeddings via Next Embedding Prediction, then gets discarded after training; the 2B model rises from 47.25 to 48.25 aggregate over 16 benchmarks with strictly zero inference overhead and 11.6% more training GPU-hours.

What problem this solves

Next-token prediction on text gives visual perception only indirect supervision: the model learns to talk about images without ever being forced to learn from them. Language has an expressivity bottleneck, so exact spatial relations and pixel-level boundaries never make it into the captions. Unified multimodal models (UMMs) try to jointly train understanding and generation to close this gap, but two old problems survive. The generation branch adds parameters that must be carried at inference, costing latency and memory. And the generation objective is optimized for synthesis quality, not understanding gains, so understanding scores often stay flat or drop.

Method

GAS follows one principle: generation serves strictly as training-time auxiliary supervision and changes nothing about the inference-time architecture. Three components make it work.

• NEP (Next Embedding Prediction) defines generation as autoregressive prediction of continuous visual embeddings. Targets are computed by the same ViT plus a projector (z = Projector(ViT(I))), landing in exactly the same domain as the LLM input space, with a normalized cosine loss. No pixel decoder and no discrete visual tokenizer are needed. The projector moves during training, so using it directly as a target would drift; the target projector is therefore maintained with an EMA (decay 0.999).
• The MoT (Mixture-of-Transformers) split forks the trunk at layer 14 (about L/2). The upper understanding layers take only text cross-entropy; a parallel generation branch, initialized as a copy, takes the NEP loss. Generation gradients flow back only through the shared lower trunk and the vision projector, pushing fine-grained visual information into the intermediate representations. The upper understanding layers stay isolated yet keep consuming those enriched representations, and are forced to exploit them under the text objective. The ViT stays frozen throughout.
• Task construction: about 10M generation samples across 5 categories and 15 subtasks (grounding, segmentation, image editing, T2I, visual CoT), built by a fully automatic pipeline. An open-vocabulary detector proposes boxes, difficulty-controlled sampling sets box size and clutter, an open-vocabulary modeler labels them, and instructions are assembled. The design rule is that generation must require deep understanding, figure out where and what to draw before generating, rather than surface alignment like draw a cat.

Training runs in two stages: first the understanding side is frozen while the generation branch aligns, then everything trains jointly with the NEP loss weight ramped linearly from 0.015 to 1.0.

Results

Two scales (2B and 4B), 16 benchmarks across four dimensions (general perception, visual reasoning, counting and spatial, video). Paired comparison at 2B:

At 4B, GAS posts the best aggregate in the unified-model family, with MMMU 57.0 and CharXiv-DS 78.4, ahead of the 7B+7B BAGEL and the 8B Emu3. Gains concentrate in the perception and spatial dimensions, exactly what the grounding and segmentation tasks target.

The mechanism controls are clean. Adding 11% more understanding data gains only 0.48. Mixing the same generation data in without supervising image outputs loses 0.11. Attaching NEP directly to the shared trunk instead of isolating it degrades broadly (Overall down to 54.95-class results). An empty MoT fork with no generation loss reaches 47.63, below full GAS. Generation data alone is not the supervision; the pair of same-domain NEP signal and MoT isolation is.

The task-relevance experiment is the sharpest: the same 200k T2I corpus with generic short prompts gains 0.24 while losing 0.75 on perception; rewriting the prompts to carry world knowledge, multi-object composition, and attribute relations gains 0.71 with all four dimensions up. Transfer follows task relevance, not sample count.

Further diagnostics back the mechanism: GAS keeps higher visual-information cosine similarity in deep layers, RefCOCO linear probes separate from layer 8 onward (ImageNet probes stay flat throughout, so the gain is region-level rather than global-semantic), and attention maps stay focused in deep layers where the baseline diverges past layer 20. Text-only reasoning is unharmed: ZebraLogic +2.8pp and MMLU-Redux +1.23pp. Three runs at 2.5B give 48.46±0.37 against a 47.25 baseline, so the gain is stable.

Why it matters

For teams training MLLMs this is a cheap, pluggable upgrade: 11.6% more training GPU-hours, strictly zero inference overhead, and it works added at any of three stages, from scratch, after pretraining, or after SFT. The task-relevance finding is directly actionable for data construction: rather than piling up generation samples, reshape the generation instructions so the model is forced into spatial grounding and compositional reasoning. For the unified-model route debate, the paper splits the understanding-generation conflict into an architecture problem (solvable by MoT isolation) and an objective problem (solvable by same-domain NEP), instead of an either-or choice.

Limitations

The gain is honest but modest: about 1 point on Overall and 1 to 3 points on most single benchmarks, far from the strong claim that generation transforms understanding. At 4B, VisuLogic actually drops from 26.8 to 23.6; the authors attribute this to the puzzle-style benchmark being scale-sensitive, but it exposes uneven transfer. The main models are trained from scratch at only 2B and 4B, with no validation on a stronger base or an aligned commercial-grade model. The 10M generated samples are fully auto-synthesized with quality guaranteed only by pipeline design and no human inspection numbers. Under the frozen-ViT setup, the finding that unfreezing the ViT hurts cannot be extrapolated to the common configuration where the ViT also trains for understanding. The paper is from ByteDance, and comparisons against unified models such as BAGEL cite public reported numbers rather than self-run evaluation, so the settings are not fully comparable.

Terms

• NEP (Next Embedding Prediction): a generation paradigm that autoregressively predicts continuous visual embeddings in the same domain as the LLM input space.
• MoT (Mixture-of-Transformers): an architecture that shares lower trunk layers and forks upper layers into task-specific parameters.
• EMA target projector: a projector maintained by exponential moving average, preventing supervision drift from a moving target.
• UMM (unified multimodal model): a model jointly trained for understanding and generation, such as BAGEL, Emu3, Janus.
• Visual CoT: a generation task that asks the model to produce intermediate visual reasoning traces (like segmentation maps) before answering.

Source

• arXiv 2608.12209v1
• PDF
• Hugging Face

Related papers

• HSImul3R cuts human-scene mesh penetration from 69.5% to 22.9% with a physics loop
• Short-Answer Hallucinations in MLLMs Start in Vision; ACFT Fixes Them on 0.9% of COCO
• LingBot-Map Hits 20 FPS on 10k-Frame Streams, Beating Offline 3D Models on Oxford Spires
• GRF-Recon hits KITTI 7.18 m ATE, matching calibrated SLAM without intrinsics
• EliGSiR beats SplaTAM by 2 dB on TUM by spending compute where the map is weak
• AMB3R-SLAM cuts VBR ATE 72% with a hierarchical backend on one GPU

All paper explainers