Autoregressive Mosaics: Probing 2D Spatial Reasoning in Text-Only Language Models
Ashwin Nedungadi, Stefan Oehmcke, Stefan Lüdtke
cs.AI, cs.CV
2026-08-31
AM-Bench splits coding from composition: eight text-only models hit median PIoU 1.0, yet layout runs from GLM-4 32B at 3.57 to CodeLlama at 1.75; SVG adds +0.37.
GPT-4 can emit TikZ that looks like a unicorn, and the model has never seen a pixel. That result does not say whether a 2D layout lives inside the network, or whether the model is only translating a spatial description into code. Recognition benchmarks such as DORI and 3DSRBench need an image as input. Generative evals score the final picture and mix composition with coding skill.
AM-Bench, from the University of Rostock, splits those two. A translation task writes the geometry out in words; the model only has to emit code. A layout task names an object with little spatial detail; the model has to compose the arrangement.
An image is an autoregressive mosaic: the model writes a Python function using six primitives (fill, setpixel, rect, circle, line, poly), rendered to a 24x24 raster by a deterministic executor. SVG is not the main interface, because public SVG is already in pretraining and would mix memory with composition. A 30-name palette keeps color tokenization from dominating.
Translation: 145 reference geometries, plus 100 harder items and 13 hand-drawn icons. Prompts list position, size, and color, and withhold object names. Scoring is per-part IoU (PIoU) in continuous coordinates, times an over-paint penalty, so the number does not depend on canvas resolution. Pass threshold τ=0.6; a random baseline sits at 0.072. A model enters the layout ranking only if its median translation score clears that line.
Layout: 150 prompts in three tiers. T1 elemental (filled shapes, outlines, tilings, symbols, color partitions), T2 iconic (flags, cultural signs, everyday objects, living things, structural patterns), T3 compositional (spatial relations, multi-object, nested, scenes, symmetry). Each prompt is sampled once greedily and ten times stochastically, 13,200 attempts across eight models. Two VLM judges, Qwen2.5-VL-7B and InternVL3-8B, score prompt fidelity, shape, color, space, and completeness from 0-5; 3.0 is a pass.
The eight open text-and-code models are Qwen2.5-Coder 32B/14B, Gemma 2 27B/9B, GLM-4 32B/9B, CodeLlama 34B, and Llama 3.1 8B.
Two extra tests. Medium ablation: the same prompts are answered in raw SVG, then rasterized back to 24x24 so resolution is not a gift. Representation: a ridge probe on the pre-generation residual stream predicts a 6x6 occupancy grid, targeting each model's own consensus occupancy across repeats; ΔR² is against a TF-IDF text baseline. A causal follow-up edits coordinates in the code prefix to see whether the model executes a plan fixed in advance or tracks geometric state already written in the text.
Translation is not the bottleneck. Median PIoU is 1.000 for all eight models; 98.1% of 13,920 attempts pass. Llama 3.1 8B is the weakest on the mean at 0.929 and still clears the bar. The models can write the code.
Layout scores spread widely. Under Judge 1, GLM-4 32B scores 3.57 overall, CodeLlama 34B 1.75, more than a 2x gap. Judge 2 is about 0.90 points more lenient, but the model ranking is identical (Spearman ρ=1.0). Per-sample agreement is only 0.43-0.54 Pearson. On 350 pairwise human comparisons, annotator Krippendorff α=0.60. Where both annotators agree and the judge separates the pair, Judge 1 matches humans on 82.4% (61/74) and Judge 2 on 76.7% (46/60).
The expected difficulty ladder fails. T3 compositional outscores T2 iconic (Judge 1: 2.69 vs 2.45). The authors blame 24x24 primitives: real-world icons need fine structure, while compositional prompts are mostly spatial language. Easiest subcategory: 1E color partitions (Judge 1 3.93). Hardest: 2C everyday objects and 3D scenes. CodeLlama emits trivial near-uniform fills on 26.9% of T1 attempts; Gemma 2 9B hits 20.5% at T3.
SVG raises every model, pooled +0.37 (95% CI [0.26, 0.47]), significant for six of eight, largest for CodeLlama (about +0.76). Relative order is similar. Absolute scores are a property of the canvas API, not of spatial reasoning in the abstract.
A coarse layout is linearly readable before generation, but it is the layout the prompt implies, not the layout this model will draw. Mean ΔR² across eight models is +0.164. On GLM-4 32B and Gemma 2 27B, the shared component is ΔR² about +0.26 / +0.28; the model-specific residual has R²≤0.005. After moving an already-drawn shape into a region the original generation left empty, GLM-4 32B continues drawing 84.6% of the time versus 43.8% (+0.42), Gemma 2 27B 41.2% versus 2.5% (+0.39). Moving it onto a region the original later fills produces a much smaller bump (+0.08 / +0.16). The model tracks textual geometric completeness. It does not execute a floor plan fixed before the first primitive.
| Model | Translation PIoU | Layout J1 overall |
| GLM-4 32B | 0.993 | 3.57 |
| Qwen2.5-Coder 32B | 0.994 | 3.44 |
| Llama 3.1 8B | 0.929 | 2.03 |
| CodeLlama 34B | 0.984 | 1.75 |
A model that can draw is not automatically a model with a spatial world model. Passing translation only says the code interface is usable. Layout spread, the medium ablation, the probe, and the intervention together point to a coarser picture: the prompt supplies a shared sketch, and the actual marks are chosen autoregressively as tokens land.
For programmatic drawing, CAD, or SVG agents, the interface moves the score. A custom canvas API is the cleaner test; SVG inflates it. If a benchmark only grades the final picture, memorized SVG templates get counted as spatial skill.
The authors say so: only 8B-34B open models, no closed or larger ones; a 24x24 mosaic is a necessary condition for spatial competence, not spatial intelligence in general; VLM judges add variance; the Exp. 3 intervention edits code text, not activations, so it cannot prove that the decoded representation is what drives generation. The shared/specific split covers only the two strongest models.
Other gaps. The 150 layout prompts were synthesized by Claude Opus 5 and may favor some families. The T2/T3 inversion is blamed on resolution, but there is no larger-canvas control, so that account is still a hypothesis. Judge ranking is stable; per-image scores are not. A DINO concept-consistency analysis is confounded by token entropy and lives in the appendix.