A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
Jennifer D'Souza, Fahad Ahmed, Cecilia Andrea Bustamante Andrade, Lina Frolova, Poorani Gnanasambandan, Dilshad Hussain, Muhammad Uzair Khan, Nkembeng Kevin Nkengfoa, Paul Praveen J., Fabio Priante, Sjoerd Franciscus van der Werf, Thomas Frederik Jan van Roeden
cs.AI, cs.CV, cs.DL
2026-08-14
TIB and partners release ALD/E-ImageMiner: 1,951 expert-annotated figures from 205 papers across 49 categories, probed by four tasks (classification, table extraction, summarization, VQA), plus an ICDAR 2026 competition and a roadmap toward scientific conceptual understanding from images.
Figures are where experimental evidence actually lives: spectra, micrographs, process schematics, multi-panel composites. Digital libraries can only index them through captions, and multimodal models read them poorly. Existing evaluations agree on the pattern: models recognize axes and legends and handle direct numerical extraction, then fall apart on spatial reasoning, cross-modal synthesis, and multi-step inference. On real materials-science scatter plots, extracted tables come back incomplete and numerically inaccurate, and performance drops as figures get denser.
Current chart benchmarks do not close the gap. ChartQA's 20,882 charts are web-sourced economic and survey visualizations, a different visual language from spectra, reaction schemes, and apparatus diagrams. FigureQA, DVQA, and PlotQA are programmatically generated. The closest materials effort, PolyCompChartIE, covers chart-to-table on scatter plots only. What is missing is a domain-grounded benchmark with complementary tasks on the same scientific figures.
ALD/E-ImageMiner is built on atomic layer deposition and etching (ALD/E), a class of cyclic surface-mediated processes for controlling film growth and removal. The domain choice is deliberate: every figure is tied to process conditions, precursor pulses, purges, growth per cycle, so reading them requires domain knowledge, not just vision.
The benchmark holds 1,951 figures from 205 experimental and simulation papers, manually sorted into 49 categories spanning line charts, spectra, scatter plots, heatmaps, molecular-structure and reaction diagrams, band and apparatus diagrams, process flows, and composite panels. MinerU first extracts structured text and high-resolution figures; annotators then manually assign each subfigure a panel letter and machine-readable bounding box (x, y, width, height), so every annotation pairs with a specific image region instead of asking the model to infer the target from the full composite.
Four tasks form a progression:
The VQA design is scaffolded on Bloom's revised taxonomy, walking from Remembering up to Evaluate through four question families: process-oriented (cyclic steps and condition changes), comparative/trend (how temperature, pulse length, cycle count shape outcomes), structure-property (how precursor chemistry and film structure determine properties), and application/performance (what observed behavior means for a device). Answers come in four formats: yes/no, factoid, list, paragraph. Paragraph answers exist so the benchmark can score whether reasoning is scientifically coherent, not just whether the conclusion is right.
The worked example is a five-panel figure of PbI2 grown epitaxially on sapphire, with four questions spanning three families and three answer formats. "Can PbI2 be grown epitaxially over sapphire" is a yes/no check; "why was the pole figure measured at 25.95 degrees" demands a full crystallographic epitaxy argument; "what suggests disorder in the films" expects the FWHM values of 13 and 9 degrees from two scans. One figure, several grades of depth.
This is a perspective paper and carries no system scores; the competition report in the same proceedings holds them. The hard numbers here are the benchmark itself:
| Item | Value |
| Figures | 1,951 |
| Source publications | 205 |
| Figure categories | 49 |
| Tasks | 4 (classification / extraction / summarization / VQA) |
| Question families | 4 (process / trend / structure-property / application) |
| Answer formats | 4 (yes-no / factoid / list / paragraph) |
For context, ChartQA holds 20,882 web-sourced charts and PolyCompChartIE and MetalThermoChartIE restrict to chart-to-table on polymer and metal-property scatter plots. ALD/E-ImageMiner is the first materials-science benchmark running four tasks on the same figures with panel-level coordinates, released on Hugging Face with one Codabench competition page per task.
For multimodal evaluation work, this is a ready-made domain stress test. Synthetic chart benchmarks are near saturation; scientific figures are the next hard problem, and panel-level coordinates let you localize exactly where a model fails.
For materials and chemistry practitioners, table extraction maps directly onto literature digitization: turning decades of spectra and process plots into searchable structured data. The paper pushes further toward retrieving experiments under comparable conditions, comparing fabrication routes across material classes, and building processing-structure-property networks.
The roadmap is the most substantive part for agent-evaluation designers. A hypothesis-formation track asks a system to generate competing explanations, pick the measurement that best discriminates among them, and revise when new evidence lands. A counterfactual track removes panels, alters plotted values, swaps labels, and checks whether the model follows the evidence or defends its old answer. A shadow challenge would hand an agent a research question derived from a held-out study plus a figure-rich literature collection and have domain experts assess the resulting proposal. These hit precisely where current agents struggle: case studies show agents complete substantial research engineering while failing at experimental judgment, evidence selection, backtracking, and telling whether a question is actually resolved.
The authors position this as a perspective: no system results here, all benchmark detail and competition numbers live in the companion report, so this paper alone gives no model-versus-model comparison.
A methodological caveat they concede: Bloom's taxonomy is a question-design scaffold, and question families do not map rigidly onto Bloom levels; the same figure can support a factoid question and an analytical one. The claimed cognitive progression is therefore approximate, carried by family and answer-format combinations rather than per-question level labels.
The binding constraint is annotation cost. 1,951 figures, 49 categories, panel-level coordinates, and four question families took twelve authors across eight institutions. Every roadmap item, cross-document synthesis, hypothesis evaluation, the shadow challenge, needs heavier expert annotation and expert grading. Whether the roadmap proceeds incrementally depends on sustained investment, not on technique. The Create level currently exists as one hypothetical example outside the released annotations, which confirms that step has not landed yet.