ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts
Liyang Fan, Chi Wei, Yitai Li, Xinping Bi, Guhong Chen, Chenghao Sun, Haoxiang Yang, Qingwen Li, Kai Yan, Hong Li, Bo Li
cs.CL, cs.CV
2026-09-16
ReFigBench rebuilds 1,000 arXiv overview figures as editable PPTX. Same GPT-5.5 specialized workflow: +3.2 in Codex, −2.2 in Claude Code; native connectors median 0.
Multimodal coding agents are now asked to turn pictures into files you can actually edit. They run inside a harness such as Claude Code or Codex, the tools, context manager, and runtime around the model. Most evaluations still isolate short tool calls, API traces, or screenshot resemblance. A low score cannot say whether the model misread the figure, planned poorly, or was failed by the stack around it.
Scientific overview figures make the gap concrete. Boxes are modules, dashed edges are gradient paths, repeated glyphs are shared weights, and the public asset is usually a raster. Reconstructing that as one editable PowerPoint slide means matching pixels and recovering native text boxes, shapes, connectors, and groups. Screenshot-only grading lets a full-slide paste through. Object-count grading lets a reversed arrow through. ReFigBench asks which layer broke: perception, agent skill, or the harness.
The contract is strict. The agent sees only the mounted source image. It must return a single-slide PPTX plus the program that built it. A file that will not open, has extra slides, or will not render scores zero.
Two workflows. Direct generation writes a program against a general PPTX library. The specialized path uses a PPTX-centered document toolchain and can render intermediates against the source. Claude Opus 4.6, MiMo-v2.5, and MiniMax-M3 run in Claude Code. GPT-5.5 runs in both Claude Code and Codex, both at xhigh reasoning. That yields ten configurations, 1,000 figures each, 10,000 artifacts.
ORBIT builds the set from cs.AI, cs.CL, cs.CV, and cs.LG papers since June 2017, ranking figures by caption evidence for overviews. A strict gate accepts 5,146 figures; a manual audit finds 98% are real overviews. The released 1,000 are a reproducible shuffled prefix from 1,000 distinct papers. Median decoded area is 1.50 megapixels, across 980 resolutions, with no shared canvas. cs.CV alone contributes 464 figures.
After the gate, GPT-5.4 scores five axes: text 20, semantic structure 30, layout 15, editability 25, visual detail 10. A near-full-canvas raster paste with few native objects is capped at 50. The primary quantity is the paired workflow effect, specialized minus direct, with the model and harness held fixed. Blinded humans pick A, B, or a tie; a Bradley-Terry model reports Elo with mean 1,000. Qwen3.6-27B, a second judge family, rescores the full suite three times.
Every configuration passes the artifact gate, so the gate does not rank systems. Mean scores from the first GPT-5.4 run:
| Setup | Direct | Specialized |
| GPT-5.5 + Codex | 74.2 | 77.4 |
| GPT-5.5 + Claude Code | 75.6 | 73.4 |
| Claude Opus 4.6 + Claude Code | 69.4 | 66.4 |
| MiniMax-M3 + Claude Code | 66.1 | 66.1 |
| MiMo-v2.5 + Claude Code | 63.4 | 61.0 |
The same specialized investment raises GPT-5.5 by +3.2 inside Codex (bootstrap interval about +2.5 to +3.9) and lowers it by −2.2 inside Claude Code. Under a word-for-word identical direct prompt, GPT-5.5 scores 1.4 points higher in Claude Code than in Codex, interval [0.7, 2.2]. Model identity still separates the field by more than 16 points; the four GPT-5.5 rows take the top four. Even the leader recovers only about four fifths of the text credit and less of the visual-detail credit. Intermediate rendering pays back part of the perception deficit, and only part.
The object audit is harsher. Under the specialized workflow, median native connectors are 0 in every configuration, and 91.8% to 100% of slides have none. Direct reconstructions keep medians between 15 and 42. Editability falls in all five pairs. Semantic failure, less than half the structure credit, exceeds a quarter of artifacts for the weakest setup and drops to 0.5% for GPT-5.5 specialized in Codex.
The bill matches the traces. Specialization multiplies median turns by about two to three in four of five pairs and raises per-artifact cost by 1.4x to 3.5x. Only GPT-5.5 in Codex converts the extra spend into a higher score, at about $3.57 per artifact, the most expensive row. Generating the suite cost roughly $15,000. Under a fixed budget, direct is the better default in four of five pairs.
Blinded human Elo correlates with rubric means at Pearson r=0.958. Humans still prefer the specialized rendering in four of five pairs, including two pairs where the rubric goes the other way: the rubric charges for stripped native structure, people reward pixels. Qwen3.6-27B is more lenient, reproduces the ranking at Spearman 0.988, and even keeps the fine order that Claude Code direct beats Codex direct. On margins above ten points, judge direction agrees with humans 91% of the time; humans agree with each other 59%.
One GPT-5.5 Codex direct case makes the remaining failure mode obvious. The visual vocabulary is almost perfect and nearly every arrow is reversed. Editability gets 22 of 25; semantic structure gets 10 of 30. The paste cap fires on 2 of 10,000 artifacts. The live risk is an editable slide that computes the inverse of the paper.
If a document-agent report names the model and hides the harness and workflow, this benchmark says that report described half a system. A specialized toolchain can lift perception (text rises in all five pairs) and still turn documents into drawings. Humans buy the render; the rubric fines missing structure. Those two scores have to stay separate.
For anyone shipping this, writing the object tree with python-pptx is still the stabler default. The specialized path is worth the extra bill only inside a harness that can cash intermediate renders for points, Codex in this study. The ceiling is 77.4 of 100. Better pixels alone do not reconstruct a scientific overview. Structure has to survive with them.
The task is locked to a single PPTX slide. ORBIT samples without domain quotas, so cs.CV is nearly half the set. The harness contrast covers only GPT-5.5. The paired delta bundles libraries, prompts, and toolchains, so it does not isolate which layer moved the score. The primary judge shares a vendor family with the strongest agent. Repeated runs, a second family, and blinded humans bound that risk at the system level; single-task scores still jitter by about three points, which is why margins under two are treated as ties.
Rubric weights remain a free parameter. The Codex +3.2 flips sign only when editability approaches half the rubric. Humans and the rubric part ways exactly where fidelity and editability decouple. System ranking is stable. Close pairs are not.