MaLiang-Harness: A Programmable Path to Image and Video Generation
Haoyu Zhao, Zihao Zhang, Xudong Wang, Jiaxi Gu, Zuxuan Wu, Yu-Gang Jiang, Shuicheng Yan
cs.CV
2026-09-28
MaLiang-Harness runs MLLM image/video generation as a construct-inspect-revise loop. GPT-6-Astra fully succeeds on 50 image tasks (48/50 all-pass); two AA-37 Lunas split 44% vs 88%.
High-quality image and video models still mostly paint pixels. GANs, diffusion, and flow matching learn a visual distribution and emit an image whose construction you cannot inspect object by object. Multimodal LLMs can instead emit an executable visual program and hand it to Canvas, SVG, Scene2d, or Three.js. Layout, timing, and camera become code.
Runnable code can still yield the wrong picture. An object can land in the wrong place; an event can fire at the wrong time. The paper names this mismatch the Program-to-Visual (P2V) gap. Closing it means the model has to look at the render, edit the program and the plan, and keep the artwork alive across rounds so a fix does not wipe the last good version.
MaLiang-Harness, from the National University of Singapore, Fudan University, and Tencent, folds image and video into one construct-inspect-revise loop. An image is the program evaluated at a content time; a video is the same program sampled over time. When appearance is hard to draw in code, the run can attach user images, search results, or generated assets.
Three pieces share one revision index.
The MLLM scores used for delivery are self-assessments, not an independent perceptual test.
MaLiang-IBench has 50 text-to-image prompts and 11 closed-source models; MaLiang-VBench has 13 text-to-video prompts and four models. GPT-6-Sol grades on a 1-5 scale: alignment, aesthetics, composition, plus motion coherence for video. A dimension counts as met at ≥4. Failed runs drop out of quality means. DeepSeek-V4-Pro ran without visual feedback.
Execution success and visual pass are different counts.
| Model | Success | All criteria | Time/Qualified |
| GPT-6-Astra | 50/50 | 48/50 | 3.70 min |
| GPT-6-Sol | 48/50 | 46/50 | 3.52 min |
| GPT-5.6-Sol | 48/50 | 43/50 | 3.28 min |
| GPT-5.6-Luna | 46/50 | 22/50 | 4.49 min |
| Kimi-K2.7-Code | 20/50 | 12/50 | 29.84 min |
| DeepSeek-V4.1-Flash | 6/50 | 6/50 | 27.15 min |
GPT models succeed on 92% to 100% of image tasks; DeepSeek and Kimi sit at 12% to 40%. All successful GPT-6 images clear aesthetics and composition; leftover misses are almost all prompt alignment. Kimi-K3's 4.56 alignment mean rests on nine images and does not rank it over the full set of 50. Early failures shrink wall-clock totals, so DeepSeek and Kimi cost figures are not evidence of thrift.
Video is steeper. Astra completes 13/13 and meets all four thresholds on 10/13 (76.9%); motion is the tightest gate (aesthetics 13/13, motion 10/13). GPT-5.6-Sol completes 7/13 and fully passes 5/13 (38.5%). Kimi-K2.6 finishes three videos, none of them full-pass. DeepSeek-V4.1-Flash (64K) fails all 13, five of them by burning the token budget.
Against Artificial Analysis Intelligence Index v4.3.2, the Spearman rank correlation with drawing full-pass rate is ρ=0.65 (n=11). GPT-5.6-Luna and GPT-6-Luna both score 37 on that index and full-pass 44% vs 88% of IBench, a 44-point gap. Kimi-K3 scores 44 on the index and full-passes 18% of tasks.
A general leaderboard is a weak picker for this job.
For posters, flowcharts, storyboards, and mixed-style clips, the offer is editability and a paper trail: layout and motion live in code, you patch the program rather than pixels, and you can roll back. In the six-panel research figure of Fig. 10, code draws the layout and plots; generated photo assets are tried four times and crops from the third attempt are placed; the numeric panels pass, the photo cells fail, and the run keeps draft status.
The benchmark measures whether a model can turn spatial and appearance constraints into executable structure and then use a render to revise. That skill is separate from knowledge and reasoning exams. The 44-point Luna split is already enough reason not to pick a drawing model by AA score.
This is a harness and a test suite, not a new generator. The backends are still 2D/3D drawing APIs; photorealism is unsolved here. GPT-6 variants sit at the top under a GPT-6-Sol judge, so cross-family ranking should be discounted.
A path-tracing backend can produce glass transmission and contact shadows, and raising samples from 512 to 1024 cuts noise, but a local geometry edit introduces overlap artifacts. Asking REV for finer Canvas brushwork thins grass cover and stone marks instead of adding realism; the result stays stylized. Refinement also stalls: reasoning stagnation, semantic livelock, and infinite agentic loops. PEG, TGP, and REV keep state and traces; they do not make the next edit work. In Fig. 8 the run inspects until the token budget runs out and never issues a corrective edit. Stopping is a budget on calls, tools, time, and tokens; finishing is the delivery predicate. Those are separate.
The eval is thin. Fifty image prompts and thirteen video prompts, all closed-source models, with budgets and feedback that are not uniform. Quality means are computed only on successes, so low-success models are not comparable on those means. Three timestamps do not prove temporal continuity. The judge is GPT-6-Sol while GPT-6-Astra, Sol, and Luna are on the list; same-family grading bias is not separately reported. The public AA score for DeepSeek-V4-Pro uses checkpoint 0813; the checkpoint in the run is unconfirmed.