Lang-SVG: Hierarchical Image Vectorization with Language Priors
Xi Liu, Chaoyi Zhou, Run Wang, Jiaang Li, Feng Luo, Junxiang Huang, Siyu Huang
NeurIPS 2026
cs.CV
2026-11-12
Lang-SVG builds semantic SVG part trees. At 576 points on ADE20K, part recall rises from LIVSS 28.8% to 48.5%, StructXLIP from SGLIVE 0.462 to 0.506.
Turning a photo into SVG means writing path code that stays sharp at any scale and that a language model can read. DiffVG made vector rendering differentiable. LIVE adds primitives one layer at a time. SGLIVE and LIVSS then take masks from SAM, a promptable segmenter, so shapes follow image regions more closely. The picture can look right while the file stays flat. The window, tire, and body of one car land on unrelated paths, and there is no handle for the car or for the door alone.
A pixel loss does not choose where control points go. With no hierarchy, fitting drifts toward high-frequency texture and leftover residuals, and a complete part can be under-drawn. LIVSS imposes levels by simplifying the photo with a diffusion model before SAM. Diffusion shifts boundaries. Masks in SGLIVE stay at a coarse object level and do not reach parts. Usable SVG of natural images is still scarce beside icon sets, so an editable intermediate representation is missing.
Each node stores a semantic tag, a mask, and Bézier primitives: smooth curves set by control points, forming one closed region. A parent-child edge means the child is a piece of the parent. The root is the full image.
The tree is nested out of segmentations. Grounded-SAM, an open-vocabulary segmenter, supplies labeled objects and parts. Semantic-SAM adds finer masks at several granularities, initially tagged unknown. Insertion runs coarse to fine. A mask may parent another only when it covers enough of the child's area, above a threshold τ. The deepest valid parent wins. Ties break on IoU, the overlap of the two regions. If nobody qualifies, the node attaches to the root.
After alpha compositing, nodes near the root are mostly hidden by their descendants, yet their outlines are the most jagged. Depth-adaptive simplification raises polygon tolerance near the root and lowers it on leaves, so simple shapes sit upstairs and boundary detail stays on the leaves. The point budget follows how much of a node you can still see.
Same-color regions with different shapes survive an IoU prune. Tree-guided pruning and merging compares SVG colors inside one semantic branch. An unlabeled child too close in color to its parent is dropped. Parent-child or sibling nodes merge when color matches and overlap is large. Nodes that already have tags are kept, so a same-colored car and road do not collapse into one shape.
Holes in the initial masks are filled by L1-SAM-Prompting. The L1 map between the current render and the photo marks wrong pixels. Turning those error clusters straight into paths pours points into texture. Each cluster's box and mean point is sent to SAM instead, and the new mask is inserted between a stage that fits the segmentation structure and a stage that cleans appearance.
Unknown nodes go to Qwen3.5-27B. The prompt carries the full image, a crop, the fill color, and the tags and boxes of ancestors and labeled relatives. Closed-set mode picks from a list. Open-vocabulary mode writes a free-form tag. Uncertain nodes become texture or unknown.
The main set is 200 ADE20K-Part234 validation images and 752 ground-truth part and subpart masks. Budgets match control points. LIVE and DiffVG use nine points per path. LIVSS and Lang-SVG do not. Same points is the main protocol, at 576, 1152, and 2304 points. Avg.R averages part recall at IoU 0.3, 0.5, 0.7, and 0.8. StructXLIP, trained with edge cues, scores structure that PSNR misses.
| Method | PSNR | StructXLIP | CLIP | Part Avg.R |
| DiffVG | 21.81 | 0.290 | 0.597 | 14.9% |
| LIVE | 21.28 | 0.420 | 0.625 | 16.9% |
| SGLIVE | 21.38 | 0.462 | 0.634 | 17.7% |
| LIVSS | 19.50 | 0.379 | 0.609 | 28.8% |
| Lang-SVG | 21.88 | 0.506 | 0.651 | 48.5% |
At 576 points the pixel margin is thin. PSNR is 0.07 dB over DiffVG (21.88 vs 21.81). SSIM moves from SGLIVE's 0.633 to 0.638, LPIPS from 0.464 to 0.436. StructXLIP moves from SGLIVE's 0.462 to 0.506, and Avg.R from LIVSS's 28.8% to 48.5%. Recall at IoU 0.5 goes from 33.4% to 60.5%, and at 0.7 from 19.7% to 37.2%.
At 2304 points Lang-SVG reports PSNR 25.90, StructXLIP 0.775, and Avg.R 64.2%, against LIVSS at 24.12, 0.695, 49.1% and SGLIVE at 25.04, 0.731, 22.3%. Matched at 64 curves, Avg.R is 51.5% versus 38.3% for LIVSS.
Dropped into LIVSS, the same Semantic-SAM and Grounded-SAM priors move PSNR only from 24.12 to 24.23 and StructXLIP from 0.695 to 0.708. Lang-SVG with those priors stays at 25.90 and 0.775.
Ablations use 2304 points. Removing the tree, which also removes pruning, drops PSNR from 25.71 to 24.29 and StructXLIP from 0.769 to 0.705. Turning off depth-adaptive simplification drops PSNR from 25.90 to 24.84 and StructXLIP from 0.775 to 0.714. Removing L1-SAM-Prompting costs 0.49 dB (25.41) and leaves StructXLIP at 0.762. UnSAMv2 in place of Semantic-SAM leaves PSNR at 25.73.
Hierarchy scores have no baseline. At a 0.5 threshold, child containment (how much of the child lies inside the ground-truth parent) is 95.55%, tagging reliability is 68.25%, and labeled subpart recall is 47.06%. At 0.7 those are 86.65%, 46.62%, and 27.00%. Containment holds. Correct labels fall off once the overlap test gets strict.
Tag recovery builds trees from ground-truth masks, hides up to 50% of labels, and asks Qwen3.5-27B to restore them from the closed set. Image plus crop scores 58.5%. Parent-object tags reach 67.7%, sibling and child tags 68.9%, relative geometry 80.2%, and color 81.3%. With every image removed and only the tree left, accuracy is still 60.7%. Geometry does the heavy lifting.
On COCO-AOKVQA at 576 points, PSNR moves from DiffVG's 20.35 to 20.53. Text-to-image StructXLIP is 0.188 for DiffVG, 0.234 for LIVE, and 0.270 for Lang-SVG. On 40 complex Iconfont images, PSNR moves from SGLIVE's 28.62 to 29.28.
What gets measured is whether a rendered region lands on a human-labeled part. A person's edit time is not measured. At 576 points, part alignment is about 1.7 times LIVSS and more than twice LIVE, DiffVG, and SGLIVE.
Selecting a part by name is where the tree beats a flat path list. Closed-set recovery from 58.5% to 81.3% comes from relatives' boxes and tags, so the hierarchy carries semantics of its own. The pipeline is test-time optimization: segment, fit a differentiable SVG, then call a 27B model. It is not one forward pass. Latency and compute are unreported.
The renderer is the same Bézier-plus-differentiable-rasterizer stack, and the pixel gain is incremental. The protocol is the piece worth keeping. SVG comparisons should report part recall along with PSNR.
Initial tags depend on open-vocabulary segmentation. Missed parts, ambiguous masks, and incomplete labels enter the tree, most visibly on small subparts and texture-like regions. Qwen3.5-27B is not trained for SVG. The stated next step is to fine-tune a multimodal model on trees and part annotations.
The 81.3% figure uses ground-truth masks and closed-set recovery of hidden labels. Segmentation error and open-vocabulary generation sit outside that test. Full-pipeline tagging is qualitative, on Pascal-Part: one region called bird tail, a smaller piece called texture.
Hierarchy scores have no comparator. Containment reaches 95.55% at a 0.5 threshold, while labeled subpart recall is 47.06% and drops to 27.00% at 0.7. Editability is the motivation, and there is no user study, no runtime, and no cost for the 27B calls. The sample covers 200 scenes and 40 icons. The main text never states τ, the color cutoff, or how points are reallocated during optimization.