CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions
Tsung-Han Wu, Heekyung Lee, Anya Ji, Haoming Chen, Trevor Darrell, Joseph E. Gonzalez, David M. Chan
EMNLP 2026 Findings
cs.CL, cs.CV
2026-08-29
Berkeley's CoVA-SFT (51.9K traces, 222K images) doubles interleaved CoT to 38.2%. Graph hits 62% over text-only; math falls to 8.5%; still trails Qwen3-VL-Thinking at 51.2%.
Text chain-of-thought works when the intermediate state is already linguistic: proofs, code, algebra. Chessboards, mazes, kinship graphs, and timetables are a different kind of problem. People sketch. Models get forced to dump the board into prose and re-list adjacency at every step. The trace gets long. The state gets lost.
Most interleaved-reasoning corpora start from an image that is already in the prompt. Zebra-CoT grounds observations in the input photo. Math-VR draws math-specific plots. The missing piece is a text-only question where the model has to build a visual workspace from scratch and keep it consistent across many steps. CoVA-SFT, from UC Berkeley, is a dataset built for that gap, with a held-out companion benchmark called CoVA-Bench.
The training set has 51,904 trajectories and 222,046 intermediate images, spanning five abstraction families and 17 tasks. The five families match how people externalize state: tables for constraints, family trees for kinship, graphs for relations and flow, layout grids for space and packing, geometry plots for math. The mix is skewed. Math-VR alone is about 32.4%, maze 7.8%, bin packing 7.2%. The rest covers chess, Sudoku, ARC-AGI, CLUTRR, graph coloring, GraphInstruct, ProsQA, seating, SpatialEval, 3D cubes, MathNet, ACP, logic puzzles, and scheduling. Some items are generated by constraint solvers (seating, zebra puzzles, 3D cubes); the rest are rewritten from public datasets.
CoVA-Bench holds out 100 items per task, 1,700 in total, and is built to test multi-step logic plus visual-state tracking.
Claude 4.5 Sonnet synthesizes each trace in three loops. It first writes why a visual scratchpad helps and what to draw, then the QA pair. It then emits interleaved reasoning, calling Matplotlib at each step and feeding the image back as context. A verification pass checks the drawing against the problem and redraws on structural errors before the next step.
The validation baseline fine-tunes Qwen3-VL-8B-Thinking on this corpus. The dataset stores real rendered pixels, but the student is not trained to call a drawing tool. Each image is packed into 128 visual tokens; the model must predict that latent trajectory plus the final text answer. Text uses ordinary autoregressive cross-entropy. Visual tokens use cosine similarity against the ground-truth embeddings. The two losses are added with weight 1. The vision encoder stays unfrozen. Training runs 2 epochs at 5×10⁻⁵, global batch 16, max response 32,768 tokens, on 8 H200 GPUs with FSDP2 and bfloat16. Baselines are all 7–8B, zero-shot, text-only prompts. Answers range from letters and names to tables and symbolic expressions, so scoring uses Gemini-2.5-flash as an LLM judge.
On interleaved visual CoT, the CoVA-SFT baseline averages 38.2%. Next is MathCanvas at 16.8%, then CodePlot-CoT 12.9%, Zebra-CoT 11.2%, and TwGI (Thinking with Generated Images) at 0.9%. That is a bit more than a 2× gap. Tool-using renderers are weaker across the board: a bad drawing poisons later steps and is hard to undo. Internalizing the workspace at least removes that tool boundary.
Text-only CoT is still stronger. Qwen3-VL-Thinking averages 51.2%, Qwen3-Think 45.1%, Qwen3-VL-Instruct 46.0%. The visual scratchpad has not caught the best prose reasoner.
| Method | Table | Layout | Graph | Game | Math | Avg |
| Qwen3-Think | 67.3 | 69.3 | 44.5 | 19.3 | 25.0 | 45.1 |
| Qwen3-VL-Thinking | 80.3 | 76.3 | 46.5 | 25.5 | 27.3 | 51.2 |
| MathCanvas | 14.0 | 22.0 | 35.3 | 0.5 | 12.3 | 16.8 |
| CoVA-SFT baseline | 47.2 | 53.4 | 62.0 | 20.0 | 8.5 | 38.2 |
The one clean win over text is Graph: 62.0% versus 57.0% for Qwen3-VL-Instruct. Per-task, CLUTRR hits 83.6% (text Thinking 39.0%) and ProsQA 93.8% (31.0%). Maze also pulls away, 77.0% versus 50.0%. Relational structure and grid paths are exactly the cases where maintaining an adjacency list in prose hurts.
Math goes the other way. CoVA scores 8.5% against 27.3% for Qwen3-VL-Thinking. MathNet falls to 4.9%, Math-VR to 10.9%. The paper's own account: math already has a good symbolic scratchpad, so latent visual tokens are a detour. Chess is 0.0%, Sudoku 0.0%, ARC-AGI 3.3%. Those three workspaces barely learned anything. Scheduling drops from 95.0% text-only to 34.5%.
This is a dataset paper, not a new architecture. The contribution is scale and a self-check loop for "build a visual scratchpad from a text-only prompt": 52K traces, 17 tasks, public hold-out. Anyone training latent visual tokens, interleaved CoT, or unified multimodal models now has a sizable from-scratch drawing corpus.
The practical split is sharp. Graphs, kinship, and mazes are worth trying with a visual intermediate. Math, chess, and Sudoku should not be expected to beat text CoT after this SFT. The paper also suggests continuous latent visual tokens may be a better bias than discrete image generation in unified multimodal models. That claim currently rests on about five points on Graph. Treat it as a hint.
The contribution list says general text reasoning is not hurt. The main paper does not report MMLU-style numbers, so that line is an author claim, not a measured result.
Claude 4.5 Sonnet wrote the traces. The verify-and-redraw loop catches many structural bugs, but small hallucinations in the text or in the plotting code are distilled straight into the training set. Downstream accuracy may be capped by those micro-errors. The authors list this first.
Visual diversity is stuck at what Matplotlib can draw: 2D tables, topology graphs, layout grids, geometric plots. All of it is static and schematic. There are no open-world spatial simulations and no continuous robot environments. What gets learned is a symbolic diagram, not perceptual vision.
Joint text-plus-cosine training is more expensive than text-only SFT. At inference a single trace can stack eight or more visual states, so context and compute both grow. Evaluation stays on CoVA-Bench, an in-distribution hold-out; no out-of-distribution tasks are reported. The judge is Gemini-2.5-flash, with no human agreement study on structured table answers. Chess and Sudoku going to zero says this corpus did not teach those two tasks. It does not, by itself, kill the visual-scratchpad idea.