Paint-Anything: Unified Any-Color Control for Image Generation and Editing
Ji Xie, Dewei Zhou, Xinyu Huang, Zhennan Chen, Xun Wang
cs.CV, cs.AI, cs.LG
2026-09-18
Paint-Anything finetunes FLUX.2-4B on Paint-500K hex labels plus high-noise pure-color anchors, lifting ACBench-T2I from 37.0 to 68.6 and Edit from 58.9 to 75.6.
Professional design needs a 24-bit hex value on a named object, not a vague "make it bluer." Today's image models render convincing scenes, yet a prompt like #10A9FF often yields a color that looks related and sits far from the requested RGB. Prior work added dedicated color encoders, learned color tokens, or inference-time sampling and attention hacks. Those paths are hard to port across architectures, hard to cover all 16.7 million 24-bit colors, and they usually split generation, colorization, and editing into separate systems.
Paint-Anything, from ByteDance Seed with Zhejiang University and Nanjing University, collapses this into one prompt interface. Hex codes sit in <color> #AABBCC </color> spans. One finetuned weight handles both text-to-image and object recoloring.
The backbone is a rectified-flow text-to-image model. The main run uses FLUX.2-klein-base-4B (about 4B DiT parameters, about 8B including the Qwen3 text encoder). The VAE and text encoder stay frozen; only the denoising transformer is trained. Generation and editing share a flow-matching loss. Editing samples concatenate a clean source-image latent to the noisy target sequence.
Supervision comes from Paint-500K. Internal image-caption pairs go through VLM object grounding, SAM3 masks, and MeanShift clustering in CIELAB to extract a dominant sRGB hex, then a VLM rewrites the caption to bind that hex to the noun phrase. Fixed-K k-means tends to split shadows into extra labels; MeanShift adapts the cluster count. The mix is 400K T2I samples (100K single-object, 300K multi-object) plus 100K edit pairs. Edit sources are recolored by a pretrained editor; the target remains the original photo so targets carry no generative artifacts.
Real-image labels are only approximate, because shadows map one object to many RGB values. Pure-color anchors fix the low-level map: sample a 24-bit RGB, render a solid image, pair it with the matching hex prompt. Those samples train only at high-noise timesteps, with default gate tgate=0.8, so low-noise training stays on natural images. Without the gate, generation improves and editing drops; with the gate, both rise.
ACBench segments the target object with SAM3, computes mean sRGB MAE against the requested hex, and maps it to a 0-100 score: channel-average MAE of 16 or below is full credit, 64 or above is zero.
| Model | ACBench-T2I | ACBench-Edit | CompColor hex avg |
| FLUX.2-4B base | 37.02 | 58.90 | 0.38 |
| Paint-Anything | 68.58 | 75.57 | 0.79 |
| FLUX.2-dev (56B total) | 51.70 | 68.87 | — |
Relative to the 8B base, T2I rises 85.3% and Edit 28.3%. The finetuned 8B model beats the 56B sibling. On CompColor, swapping color names for hex codes drops the base average from 0.72 to 0.38; after finetuning the hex average is 0.79 and the named-color average also moves from 0.72 to 0.79. Independent GenColorBench NCU goes from 34.00 to 57.89, above the published FLUX+NumColor figure of 51.90. In 1200 judgments from 15 people on 80 T2I prompts, the finetuned model wins 55%, ties 32%, and loses 13%.
Full finetuning beats rank-256 LoRA (T2I 68.58 vs 48.45). Wrapping hex spans without pure-color anchors reaches 57.16 T2I and 73.89 Edit; adding gated anchors completes the recipe. The same recipe lifts Z-Image Base T2I from 33.45 to 53.77.
Brand colors and SKU shots used to need extra modules or inference-time optimization. Object-level hex labels plus high-noise pure-color anchors put that control back into ordinary prompts, with one weight for generation and editing. Teams can try this recipe on FLUX.2-4B instead of jumping to a 50B-class model. It is a training-recipe result, not a new generative family.
There is no palette-level supervision, so shared palettes across objects remain untested. ACBench scores depend on segmentation and region means, which smear texture, specular highlights, and translucent materials. Edit sources are model-synthesized and may not match real user photos. NumColor had no runnable inference code, so the NCU comparison quotes published numbers. The user study covers T2I only, and repeated participants and prompts prevent treating it as per-sample human-metric agreement.