VibeEdit edits images from marks drawn on the canvas, beating text-prompted baselines 79.9 vs 67.4

VibeEdit: Image Editing with Canvas Instructions

Jinjing Zhao, Fangyun Wei, Yitong Wang, Xiuyu Wu, Yunuo Chen, Yang Yue, Sirui Zhang, Wenbo Wang, Hongyang Zhang, Dong Chen, Yan Lu, Chang Xu

cs.CV, cs.AI

2026-10-09

VibeEdit treats circles, arrows, and handwritten notes drawn on the image as the edit instruction, scoring 79.9 on a 419-case benchmark, 12.5 points above the best text baseline.

What problem this solves

Instruction-based editors handle "change the color" fine and fumble on "which of the three identical chairs". The user compresses a spatial reference into prose and the model guesses the referent, so wrong-target edits are the standard failure mode. Pointing tools fix location: DragGAN drags points, MagicQuill paints brushes, but semantics still ride on a separate text prompt, splitting one edit into two inputs. VibeEdit merges them. Circles, scribbles, arrows, and short handwritten notes drawn on the image form a canvas instruction, with no separate text prompt, covering addition, removal, replacement, attribute modification, and movement.

Method

Three parts.

Data: 1.55M pairs with annotations rendered online. Florence-2 proposes candidate objects, GPT-5.6-sol picks one worth editing and writes the edit description, SAM 3 segments the mask. Removal comes from ObjectClear inpainting and reverses into addition; attribute edits and moves come from Qwen-Image-Edit; replacement from FLUX.1 Fill inside the mask. The design choice that matters: storage and rendering are decoupled. The corpus stores images, masks, and structured edit fields, and canvas instructions are rendered on the fly during training with randomized stroke shapes, widths, and handwriting or print typefaces. One edit pair yields many annotation styles, so the model learns the semantics of an instruction instead of one particular hand.

Architecture: layer-decoupled conditioning. Feed the composited annotated image into the generative pathway and strokes leak into the output. VibeEdit splits the input: a frozen Qwen2.5-VL reads the annotated image for semantic tokens, while the VAE separately encodes the clean source image and the instruction layer rasterized on a gray background as DiT conditions. The ablation prices the split: source only 24.7, annotated composite 58.6, two separate layers 67.8.

Training: region-weighted SFT, then rubric-guided RL. In a local edit most pixels do not change, and a uniform loss dilutes the region that does, so object and annotation regions get a 1.5x loss weight over a mask dilated by 50 pixels. RL follows DiffusionNFT: a VLM judge answers binary questions on edit success, outside preservation, and local quality, plus a penalty when outside-region PSNR falls below 25 dB. The 3,520 RL conditions are picked for high reward variance across rollouts, because variance is what provides the comparison signal.

Results

The benchmark has 419 human-curated cases built around picking the right target among similar objects. Text baselines get a human-written instruction averaging 21.3 words; VibeEdit gets the marked image and nothing else. FireRed is the strongest text-instructed baseline.

Numbers worth pulling out:

• RL carries the gain: 67.6 to 79.9, up 12.3, with PSNR flat.
• All six public baselines score higher with external text than with the annotated image. Current models largely cannot read marks on an image; GPT-Image-1 manages 42.7 with 13.4 dB outside-region PSNR, close to a full regeneration.
• Movement separates the field most: the best text baseline (JoyAI) reaches 54.6, VibeEdit 79.9.
• Reward ablation: success-only rewards give 69.2; all three groups give 79.9 and 32.8 dB, with preservation rewards lifting both metrics.
• Removal mark design matters: a circled scribble with "remove it" scores 81.4, without the text cue 55.4.
• Judge check: on 100 sampled outputs, GPT-5.6-sol agrees with majority-vote humans at 93.0% accuracy, Cohen's kappa 0.805.

Why it matters

For editing products, the paper shows the cost of pointing can move from language to canvas, and the model side is a LoRA adaptation of Qwen-Image-Edit rather than a retrain. Circling an object and jotting a word beats writing a spatially precise sentence.

For researchers, two reusable ideas. Storing edits and rendering annotations online applies to any interaction signal (clicks, boxes, trajectories) you want to turn into training data. Layer-decoupled conditioning addresses the general problem of instructions that must guide generation without appearing in the output.

To be clear, the components are off the shelf; the contribution is the interface, the data scale, and the combination. Incremental, but a clean increment.

Limitations

• No dedicated limitations section. The authors' one concession: data filtering leaves residual noise, which RL is meant to absorb.
• Every training target comes from an existing editor (Qwen-Image-Edit, FLUX.1 Fill, ObjectClear), so edit quality is bounded by the teachers. Evaluation runs only on the authors' own 419 cases, with no external editing benchmark.
• The eval judge is GPT-5.6-sol while data construction and RL rewards (GPT-5.4-mini) also come from GPT models, and human validation covers 100 samples. Same-family circularity is plausible.
• Joint training is not uniformly better: a removal-only model scores 81.4 on removal against 73.3 for the joint model, replacement 77.6 against 72.9.
• The instruction grammar is templated (add {object}, remove it, change {attribute} to {value}); handwriting fills slots rather than writes freely, and stroke-color robustness covers only red, blue, and yellow.
• Inference settings differ: baselines run 30 steps at CFG 4, the RL model 16 steps at CFG 1, so part of the PSNR gap may come from sampling configuration rather than model quality.

Terms

• Canvas instruction: spatial marks plus short notes drawn on the source image, together specifying what to edit and how.
• PSNR: peak signal-to-noise ratio, a pixel-level similarity score; reported outside the edit region to check preservation.
• LoRA: low-rank adapters trained on top of a frozen model.
• Rubric reward: a reward split into binary questions, each scored by a VLM judge.
• DiffusionNFT: a diffusion RL method that builds positive and negative predictions along the forward noising process without storing full denoising trajectories.

Source

• arXiv 2610.12229v1
• PDF
• Hugging Face
• GitHub · https://github.com/ZhaoJingjing713/VibeEdit

Related papers

• Lag-chosen gain cuts unguided SiT-XL ImageNet FID from 28.0 to 12.2 at NFE 25
• StreamMAE: self-supervised pretraining on 95 hours of continuous video matches i.i.d. training
• Pumpire finds depth leaderboards do not predict tape-measured distances
• Open models top out at 23% while GPT-Image-2.5 reaches 56.6%
• OneSearch-VL-8B gains 20.2 and 17.6 points on multi-image and video research
• Blurry renders beat sharp point clouds: LEGO hits 20.28 dB PSNR on Ego-Exo4D

All paper explainers