CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles
Mir Tafseer Nayeem, Susmoy Chakraborty, Davood Rafiei
cs.CL, cs.AI, cs.ET, cs.LG
2026-09-29
CineSubBench tests 9 LLMs on 8.13M subtitle lines from 1,012 films in 6 languages; even frontier models omit a major event in 79.7% of synopses, with Persian input uniformly hardest.
Law, medicine, and software engineering all have dedicated LLM benchmarks. Film never got one, even though it demands long-form narrative integration, multilingual reading, and audience judgments that shift by country. Existing long-context benchmarks mostly use articles and books, where narrative structure is written into the prose. A subtitle track is thousands of short, timestamped utterances with no speaker labels, scene boundaries, or event markers; the point of an exchange in minute three may only land two hours later, and the model has to assemble characters, relationships, causality, and themes from dialogue spread across the whole film. Full-video benchmarks fold frame selection, decoding, and cross-modal inference choices into the score, and copyright plus storage cap their scale. Subtitles sidestep all of that and hand every model, open or closed, the same text interface. CineSubBench builds on this to establish film as a long-context evaluation domain, with tasks, languages, and cultural judgments matched over the same films.
Construction starts from 6,186 films in Kids-in-Mind, joins sources through IMDb identifiers, and filters for metadata completeness down to 2,322. A coverage optimization then searches for the set of languages sharing the most films instead of picking by popularity: the winning six (English, Arabic, Indonesian, Persian, Romanian, Vietnamese) retain 1,231 films, and the same algorithm over rating systems selects ten national classifications, landing on 1,012 films. Arabic ranks only eighth in raw frequency, so greedy selection would have missed it. Every subtitle track passes layered verification covering SRT structure, timestamp syntax, chronological consistency, runtime coverage, and an encoding cascade; flags trigger manual inspection or cross-provider re-download. Out of 8.13M entries, 6,594 ads, credits, and contact details were stripped.
Seven tasks in two families. Narrative: plot premise, spoiler-aware synopsis, key message, and 22-label multi-label genre. Cultural: age suitability, ratings in ten national systems (each kept as its own ordinal scale), and subtitle-grounded language safety, where the model must output the subtitle line indices behind each profanity category, separating evidence recovery from category assignment (English track only). Nine models compete; open-ended generation is scored on a 1-5 rubric by GPT-5.6 Luna as a reference-based judge, with 98.4% within-one-point agreement against an independent judge.
| Model | Composite | Narrative | Plot | Synopsis | Genre micro-F1 | Rating EM | Safety strict F1 |
| Gemini 3.8 Flash | 78.04 | 3.54 | 4.16 | 3.34 | 84.93% | 77.87% | 77.8% |
| GPT-5.6 Sol | 74.70 | 3.52 | 4.10 | 3.40 | 81.55% | 65.42% | 74.5% |
| DeepSeek V4 Pro 1.6T | 65.52 | 3.25 | 3.91 | 2.93 | 81.63% | 58.03% | 48.3% |
Leading on the composite is not leading everywhere: Sol writes the best synopses (3.40) while Gemini takes the other six columns. DeepSeek beats Sol on genre micro-F1 (81.63% vs 81.55%) and within-one-year age accuracy, yet manages only 48.3% on safety evidence F1, 26 points below Sol.
Four harder findings:
For teams building long-context, multilingual, or safety evaluations, this is a ready-to-use text-only benchmark: inputs average 40-45K tokens with peaks above 100K, no video pipeline, no copyrighted redistribution. It separates narrative integration, cross-lingual consistency, cultural calibration, and evidence grounding into independently reportable dimensions, which is exactly what aggregate scores flatten. For film search, summarization, or catalog products, the synopsis column is a ceiling check: frontier models still cannot write event-complete storylines.
The authors scope safety assessment to lexical evidence recoverable from subtitle text; gestures and boundary-fuzzy derogatory categories are excluded. Points that hold up less well: cross-lingual comparisons fold subtitle translation quality into the score, and the paper verifies only that each track is in the right language, never auditing translation fidelity, so Persian's uniform worst-case result may partly be a translation artifact; the film inventory comes from Kids-in-Mind and skews toward its US family-guidance context, with no Chinese rating system among the ten; open-ended generation is graded by an LLM judge, and 98.4% agreement is still model grading model. The paper is a preprint.