AutoRef Finds a Harness That Lifts Frozen FLUX.2 4B from 5.72 to 7.37 on MultiBanana

AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation

Yuta Oshima, Ku Onoda, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta

cs.CV, cs.AI

2026-09-29

AutoRef rewrites the frozen-model harness via a coding agent; AutoRef-Harness lifts FLUX.2 [klein] 4B from 5.72 to 7.37 on held-out MultiBanana, matching Nano Banana Pro.

What problem this solves

Multi-reference image generation asks a model to compose one new picture from several reference images, each specifying a person, object, garment, background, or style. Current generators still drop subjects, duplicate them, or glue them on like stickers.

Recent systems wrap a frozen generator and a frozen reasoning model in a harness: executable code that says how to read the references, write prompts, sample images, diagnose failures, and pick a final output. References play different roles here, and the output has to look faithful to each one and coherent as a scene. Almost every part of the harness could be rewritten, and the effect of each rewrite is hard to guess. Hand-written harnesses on the same FLUX.2 [klein] 4B already spread out: GEMS falls to 5.37, and Idea2Img reaches 7.12 only after nine generations.

Method

AutoRef leaves both models frozen and lets a coding agent rewrite the harness. The proposer is Claude Fable 5.1 through the Claude Code CLI. Search uses FLUX.2 [klein] 4B as the generator and GPT-5.5 as the reasoner.

Visual scores are noisy, a scalar tells you almost nothing about what went wrong, and repeating the loop on a small evaluated set overfits both the tasks and the judge. AutoRef moves harness search from verifiable domains onto this perceptual one with two choices:

• Split the search tasks. 48 training tasks dump scores, rationales, traces, and images into the history the proposer can read. 48 validation tasks rank candidates only; their scores and artifacts never enter the history.
• Keep a beam. Each iteration writes 4 harnesses and keeps the top 2 on validation as parents. The proposer learns which two survived, not their validation scores. The beam starts from a single generator call plus GEMS. Five iterations evaluate 20 candidates.

The harness that comes out, AutoRef-Harness, draws three images per task and hands every other decision to the reasoner. Four pieces were added in iterations that raised the validation score:

• Reference-grounded prompting. The raw user instruction never goes to the generator. The reasoner reads the instruction and all references, writes a prompt that binds each requested element to its image, and drops content the instruction did not ask for.
• Structurally diverse drafts. Draft A describes the scene subject by subject. Draft B treats the reference named as background or style as a canvas and paints the other subjects into it. If no such reference exists, B is a second sample of A's prompt. Resampling one prompt rarely fixes the pasted-in look, so the two prompts are built differently.
• Complaint-directed revision. The reasoner lists up to five concrete complaints about the winner of A vs B, each naming a reference, then rewrites the prompt and samples draft C.
• Failure-aware selection. Hard failures are counted first: missing reference, extra or duplicated subject, wrong background. Fewer failures wins. On a tie, pairwise comparison runs in both presentation orders, and the challenger must win both to replace the incumbent. Selection uses GPT-5.5, not the MultiBanana evaluator.

In-place editing of the current draft was tried and dropped: on 12 training tasks flagged as failures, 8 still failed after the edit.

Results

The MultiBanana judge is Qwen3-VL-8B-Instruct. It scores instruction alignment, reference consistency, background-subject match, physical realism, and visual quality, then averages them on a 1-10 scale. Held-out four-reference test, n=133:

AutoRef's second-best harness still scores 7.00, above Meta-Harness and greedy search. Idea2Img drops to 6.42 when cut to the same three generations. Background-subject match moves from 5.04 to 7.20, physical realism from 5.64 to 7.77.

The same harness, with no re-search, lifts FLUX.2 [klein] 9B from 5.78 to 7.10 and Qwen-Image-Edit-2511 from 4.44 to 5.69. On unseen three-reference tasks the 4B model goes from 6.94 to 7.76; on five-reference, 5.19 to 6.36. OmniContext, never used in search and scored by its official GPT-4.1 judge, goes from 8.30 to 8.85 for the 4B model, still below GPT-Image-1.5 at 9.41. Swapping GPT-5.5 for open-weight Qwen3-VL-32B still lifts four-reference from 5.72 to 6.90, below the 7.37 with GPT-5.5.

Ablations: drop structurally diverse drafts, complaint revision, or failure-aware selection and the score falls to 7.01, 6.99, and 6.93. Grounding alone with one image scores 6.91; selection alone scores 6.15. On Qwen, the harness alone at 5.69 already beats DyRef (supervised fine-tune then reward optimization) at 4.91; stacking both reaches 5.90.

Four raters compared pairs on 50 four-reference tasks: 70% win / 13% tie / 17% loss against the 4B base model; 64/12/24 against the 9B; 63/10/27 against Seedream 4.5; 46/14/40 against Nano Banana Pro, a slight edge.

Why it matters

This is inference-time program search, not another multi-reference fine-tune. An open generator that already accepts several references can pick up the released AutoRef-Harness and gain, including when the generator, reference count, benchmark, evaluator, or reasoner differs from the search. Fine-tuning and harness search stack; the DyRef numbers are complementary, not a substitute.

The cost is three generations plus several GPT-5.5 calls per task. Closed models already sit above 7.0 with one shot. Idea2Img needed nine generations for 7.12; AutoRef-Harness hits 7.37 with three. Under a matched budget, the program structure is worth more than extra samples.

The same-family 9B model scores 5.78 in one shot and 7.10 with this harness, still below 4B plus the harness at 7.37. On this benchmark, rewriting the calling program beats swapping in the larger weights. For ad compositing and virtual try-on, the released harness is the thing to try. Re-running the AutoRef search on another perceptual task still needs a Claude-class coding agent and a task set that splits into train and val.

Limitations

The paper is clear that a harness cannot outrun the generator. Qwen-Image-Edit-2511 on five references scores 1.89 vanilla and 1.87 with the harness; better selection does not help when every draft is unusable. The search maximizes a single VLM judge.

Human preference against Nano Banana Pro is only 46% vs 40%, a thinner gap than 7.37 vs 7.20 on the automatic score. The headline run is tied to GPT-5.5; Qwen3-VL-32B lands at 6.90. The Meta-Harness baseline ran five iterations here, while the original paper often uses 20-40, so the search-algorithm comparison may overstate AutoRef. Dollar cost and wall-clock of the search are unreported. Whether Qwen3-VL-8B can be gamed by the harness's failure-check wording is untested.

Terms

• harness: Executable program around frozen models that specifies how to call the generator and reasoner and how to pick an image.
• multi-reference image generation: Compose a new image from a user instruction plus several reference images, staying faithful to each.
• reference-grounded prompting: The reasoner binds each requested element to its reference image before the generator sees a prompt.
• failure-aware selection: Count hard failures first, then run pairwise comparison in both orders on a tie.
• beam search (here): Keep the top-B harnesses on the hidden validation set as parents for the next iteration.

Source

• arXiv 2609.35530v1
• PDF
• Hugging Face
• GitHub · https://github.com/KuOnoda/AutoRef

Related papers

• Column shuffle self-pairs lift variable-length scene edits to 47.6% accuracy
• Same-time reallocation lifts HunyuanVideo sparse attention by 11.8 dB PSNR
• Frozen ALPS tokens hit 85.5% on LIBERO while matching V-JEPA 2-L
• Training-free segmentation: a frozen MLLM draws B-spline masks itself, 71.5 cIoU on ReasonSeg
• Wrong tap, same login screen: LGWM hits 0.987 AUC on GUI transitions in 17 ms
• Asymmetric distillation lifts 10-turn consistency from 3.03 to 3.85

All paper explainers