A New Benchmark for Evaluating VLM ASCII Art
East-Muffin-6472 · reddit · 2026-07-19
This is a new benchmark named ASCIITermDraw-Bench, designed to test whether VLMs can genuinely generate and edit ASCII diagrams as instructed.
The author's motivation: while many models can "describe" what an architecture or topology diagram looks like, accurately formatting that content into ASCII text requires a different, harder capability. The benchmark features 80 tasks across four categories:
- Basic boxes and layouts
- Network topologies
- Software architecture diagrams
- Image-conditioned ASCII editing (modifying a diagram while preserving unrequested parts)
The evaluation methodology is comprehensive:
- Structural Score: Checks if necessary labels, edges, entities, and relationships are correct.
- Semantic Score: Evaluated by an LLM judge, repeating each task 5 times to minimize variance.
- Final results aggregate the 80 tasks, providing a 95% confidence interval.
In the current leaderboard, Gemma-4-31B-IT leads with 73.8%, followed by Qwen3.7-Plus (70.2%), Kimi-K2.6 (61.8%), and MiniMax-M3 (59.5%). The project has open-sourced 12 sample tasks and the complete methodology, with reproducible content available on Hugging Face.
Related event: ASCIITermDraw-Bench Tests VLMs on ASCII Drawing(2 posts)→
More from Multimodal
- Midjourney style code share: --sref 2912175708 — tisch_eins · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11
- MiniMax H3 MAX nails cooking anime clips: 15-second curry demo with prompts shared — Hailuo_AI · 2026-09-11
- MiniMax Music Production Toolkit 2.5 for ComfyUI adds full mastering chain — Vivid_Promise1700 · 2026-09-11
- New Node Finder for ComfyUI ranks fresh nodes by star velocity and recency — Luke2642 · 2026-09-11
- Using a finisher move on one mosquito with MiniMax H3 MAX — the bug survives — Hailuo_AI · 2026-09-11