VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei, Song Dai, Jiayu Yang, Deyuan Liu, Xiang Zheng, Xiaoshuang Shi, Hao Cheng, Kaidi Xu
cs.CV
2026-10-01
VTR-Bench tests 11 video generation models on 300 text-heavy scene prompts: the best still hits 0.250 WER, and a keyframe-guided agent framework cuts Minimax H3's WER by 32.5%.
Video generation models now produce cinematic footage, but the text inside their scenes tell a different story. The paper opens with a Seedance2.5 clip that scores 0.90 on a 20-question checklist of scene and motion requirements, yet renders the poster heading as "Posterer" and misspells the small print, for a word error rate of 0.552. Visual polish and textual accuracy are separate skills, and mainstream benchmarks like VBench and VideoPhy measure almost exclusively the former. The few text-focused benchmarks, T2VTextBench among them, cover only short strings and depend entirely on human scoring, which makes repeated evaluation expensive. For ads, science explainers, and UI demos, wrong text invalidates the whole video.
VTR-Bench pairs a dataset with a decoupled evaluation pipeline. The organizing idea: measure "is the text right" and "is the scene right" separately.
Construction is a multi-stage pipeline. Experts define five scenario categories (advertising, science, user interfaces, culture, daily life); GPT-5.6 generates scene seeds; humans screen them; DeepSeek-V4-Flash expands each into a full prompt specifying every string verbatim plus its carrier (the object the text lives on: poster, label, screen). A feasibility loop follows: Qwen-Image renders three reference images per prompt, a VLM checks whether the text and carriers fit a coherent scene, failing prompts get revised and re-checked, and three consecutive failures trigger a rebuild before final human review. One deliberate choice: spelling errors in the reference images don't count against the prompt. The benchmark grades whether the task is well posed, and refuses to simplify itself to accommodate weak models.
The final suite holds 300 prompts (60 per category) with 1,202 annotated text blocks, 2-6 per prompt and 3-5 for 94.3% of them. Required text per video runs 58 to 496 tokens with a median of 102.5, territory earlier benchmarks never touched.
Evaluation runs on two independent branches:
Both branches were validated against human annotators, and a second evaluator (Qwen3.6-27B) reproduced the ranking of all 11 models exactly.
| Model | Overall WER ↓ | Overall Video Score ↑ |
| Wan3.0 (proprietary) | 0.250 | 0.849 |
| Minimax H3 (open) | 0.447 | 0.756 |
| Seedance2.5 (proprietary) | 0.641 | 0.791 |
| Kling v3.0 (proprietary) | 0.979 | 0.793 |
| LTX-2.3 (open) | 0.996 | 0.607 |
Apart from Minimax H3, every open-source model sits near the metric's ceiling, essentially unable to produce usable text. The most telling pair is Kling v3.0 versus Seedance2.5: both land around 0.79 on video requirements while their WERs split 0.979 against 0.641. Scene adherence and text fidelity are different dimensions, and an aggregate score hides the gap completely.
The failure analysis covers 13,213 text targets: 23.31% were "carrier found but text unreadable" versus 9.34% "carrier not found", and the dominant diagnostic words are "small" (3,595 occurrences) and "blurry" (3,196). Legible still doesn't mean correct: Kling v3.0 transcribes 79.62% of targets yet scores 0.979 WER. Raising resolution lowers Minimax H3's WER, but LTX-2.3 stays near 1.0 across all four resolution/duration settings, so configuration cannot rescue a model that lacks the underlying capability.
The paper also ships an inference-time fix, the Keyframe-Guided Agentic Framework. A Director agent (Qwen3.7-plus) iteratively generates, inspects, and edits first frames with Qwen-Image3.0 until the text is right, writes a motion plan, and conditions the unchanged video model; a VLM then checks text stability and picks the best candidate. On Minimax H3 this cuts WER from 0.4468 to 0.3015 (a 32.5% relative reduction) while raising Video Score from 0.7562 to 0.8315, and beats first-frame-only I2V conditioning by a further 9.4% (I2V wins only the UI scenario). Human alignment: 92.15% agreement on the yes/no queries, video-level WER Lin's CCC of 0.9835.
This work turns text rendering from a vibe into a trackable metric and shows it is largely decoupled from scene adherence: a rising Video Score says nothing about spelling. For anyone building ads, education, or UI content with these models, the takeaway is blunt: don't trust generated text; even Wan3.0 gets a quarter of the words wrong. The agent framework is a training-free mitigation usable today, paid for in extra image-generation and VLM inference per video. It also quantifies something previously unmeasured: Wan3.0 holds a real lead over the other proprietary models on this dimension.