A New Benchmark for Evaluating VLM ASCII Art
East-Muffin-6472 · reddit · 2026-07-19
This is a new benchmark named ASCIITermDraw-Bench, designed to test whether VLMs can genuinely generate and edit ASCII diagrams as instructed.
The author's motivation: while many models can "describe" what an architecture or topology diagram looks like, accurately formatting that content into ASCII text requires a different, harder capability. The benchmark features 80 tasks across four categories:
- Basic boxes and layouts
- Network topologies
- Software architecture diagrams
- Image-conditioned ASCII editing (modifying a diagram while preserving unrequested parts)
The evaluation methodology is comprehensive:
- Structural Score: Checks if necessary labels, edges, entities, and relationships are correct.
- Semantic Score: Evaluated by an LLM judge, repeating each task 5 times to minimize variance.
- Final results aggregate the 80 tasks, providing a 95% confidence interval.
In the current leaderboard, Gemma-4-31B-IT leads with 73.8%, followed by Qwen3.7-Plus (70.2%), Kimi-K2.6 (61.8%), and MiniMax-M3 (59.5%). The project has open-sourced 12 sample tasks and the complete methodology, with reproducible content available on Hugging Face.
Related event: ASCIITermDraw-Bench Tests VLMs on ASCII Drawing(2 posts)→
More from Multimodal
- Butter.video wins over skeptical creator with hundreds of tasteful timeline 'blocks' for video effects — beechinour · 2026-10-27
- $50, 2-hour Kurzgesagt-style AI video reignites debate over intent in AI creation — motionbynick · 2026-10-27
- Fake BTS of a dam disaster made with Dreamina Seedance for 29 cents — Standard_Spell5873 · 2026-10-27
- Redditor struggles with outfit swap: Qwen Image 2.1 breaks body proportions — Existing-Box-7866 · 2026-10-27
- Kling 4.0 lands on Deevid AI: 30-second videos, 10 keyframes, 15 multimodal references — nikola_mr64990 · 2026-10-27
- Consistent characters in ComfyUI: RealVisXL V5 + img2img face reference workflow — MiaLarsonJL · 2026-10-27