PaperBanana-Interact: Scientific Diagram Refinement with Multi-Turn Human Feedback
Xueqing Wu, Ashwin Balasubramanian, Bingxuan Li, Dawei Zhu, Kai-Wei Chang, Yale Song, Yiwen Song, Rui Meng, Tomas Pfister, Nanyun Peng
cs.CL, cs.CV
2026-08-31
292 figures and 3,518 requirements for multi-turn scientific diagrams. Baselines drift and forget; a multi-objective critic lifts quality 11.9-18.6 and cuts forgetting 3.7-6.2.
Single-turn scientific diagram generation already has systems such as PaperBanana and AutoFigure. Author preferences are hard to state in one shot. In a formative study of 14 people, everyone asked for more edits after the first draft, 86% rated the refined figure higher, and the mean score moved from 3.2 to 4.2. In 64% of sessions the first draft itself changed how they thought the figure should look. Requirements grow while looking at the image. Multi-turn workflows have barely been measured.
The benchmark comes first. MTPaperBananaBench mines design choices from expert figures in PaperBananaBench, proposes candidates with Gemini, then has humans revise them: 292 figures, 3,518 requirements, about 12 per figure, covering content, organization and visual encoding, Cohen's κ 0.767. A user simulator drives evaluation. Each turn it scores the latest figure against a hidden requirement list, takes the first k failures, and turns them into natural-language feedback, for up to 5 turns. Primary metrics are requirement satisfaction and a quality win rate against the human reference. Per-turn satisfaction and forgetting are reported as diagnostics.
Baselines split in two. Generative editors (NanoBananaPro, GPT-Image-2) edit the latest image given all user utterances. PaperBanana-DirectRefine rewrites the previous drawing prompt and re-renders. Both can follow a single request (56.2% per-turn for NanoBananaPro, 68.6% for DirectRefine) and then fall apart over turns.
PaperBanana-Interact adds three pieces. A summarizer compresses the multi-image history into a textual memory of what each version showed, what changed, and whether the change matched that turn's request. Inside a loop, a multi-objective critic writes structured notes on the current request, all prior requests, faithfulness to the source paper, and presentation quality. A refiner updates the prompt, a visualizer redraws, and the loop stops when the critic is satisfied or after 10 inner iterations.
With k=1, starting from a PaperBanana single-turn draft:
| Refiner | Quality↑ | Req↑ | Per-turn↑ | Forget↓ |
| Single-turn (no refine) | 50.3 | 25.2 | — | — |
| NanoBananaPro | 19.0 | 45.2 | 56.2 | 19.0 |
| DirectRefine | 47.1 | 52.6 | 68.6 | 18.8 |
| Interact | 61.2 | 58.0 | 77.2 | 12.6 |
NanoBananaPro drops quality from 50.3 to 19.0. DirectRefine raises requirements and still lands below the single-turn quality. Interact is the only refiner that lifts both. Drift is worse at k=3; Interact still moves quality from 50.3 to 54.5 while DirectRefine falls to 42.6. In a 150-sample human pairwise test, Interact is preferred over NanoBananaPro and DirectRefine in 81.3% and 76.7% of cases. Cutting inner iterations from 10 to 1 drops quality from 61.2 to 43.5 and raises forgetting from 12.6 to 16.8. Dropping history memory raises forgetting from 12.6 to 14.7.
The paper names two concrete failure modes of multi-turn scientific figure editing: quality drift and forgetting. A multi-objective critic plus compressed memory is a reusable fix, not tied to papers; any image editor that overwrites last turn's details can use the same loop. The simulator makes the benchmark rerunnable without recruiting 14 authors each time.
The gain is a workflow gain. The backbone is still Gemini-3.1-Pro plus NanoBananaPro.
Forgetting is not gone: the best runs still sit at 10–13%. The simulator sees the judge's failure rationales, so its feedback is cleaner than a real author. Requirements and quality both go through the Gemini family, same lineage as the generator; κ=0.812 with a human judge is close, and same-family bias remains possible. All 14 formative participants came from one organization. AutoFigure was dropped from multi-turn because its single-turn quality was too low, so the comparison set is incomplete. The tool is framed as assistive; factual correctness of published figures still needs a human.