NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
Yuheng Huang, Jianlang Chen, Jiayang Song, Hua Qi, Aza Kai, Vincent Markert, Edison Marrese-Taylor, Jianjun Zhao, Lei Ma
cs.CV, cs.AI, cs.MM
2026-08-13
NARU tests Japanese long-video narrative and cultural QA on 1,481 items over 146.8 hours. Gemini-3-Flash scores 76.2%; the best open model hits 39.8%, Flash 57.4% on subtext.
Long-video benchmarks have split into two tracks. One tests retrieval and temporal grounding, LongVideoBench and LVBench. The other tests storylines and character continuity, VRBench and StoryVideoQA. Social and cultural reasoning is a third pile, usually on short clips and mostly in English.
The hard case is both demands at once: hold characters, conflicts, and themes across 30 to 240 minutes, then read meaning that is never said. Surveillance footage can be long and still yield to local detection. Japanese drama, variety, and documentary do not. Offering another cup of coffee can be a polite way to end a visit. NARU, from the University of Tokyo, Kyushu University, and Infinimind, isolates that high-context, non-English setting.
A YouTube crawl over 16 upload categories produced more than 100,000 unique videos. ASR language ID left 51,643 Japanese clips; the LVBench cutoff of 30 minutes left 8,018. Native-speaking authors then required visual integrity and temporally ordered semantic change, dropping rain-loop videos, static frames, and unrelated compilations. Greedy sampling on title and description embeddings yielded 155 videos, 146.8 hours, each 30 to 240 minutes.
The 1,481 four-choice items sit in nine slots. Narrative (745): character or entity evolution (N.1), sequential or topical flow (N.2), plot or conflict progression (N.3), thematic development (N.4). Cultural (736): aizuchi (C.1), reading the air (C.2), subtext (C.3), cultural grounding (C.4), affective dynamics (C.5).
Annotation is a hierarchical memory pipeline driven by Gemini 2.5 Pro. Each video is cut into roughly five-minute chunks. The model writes summaries, entities, events, and captions, carrying a textual recap into the next chunk. A second pass finds chapter-level segments. Separate modules then write narrative threads and timestamped cultural evidence. Questions are synthesized from those notes, then debiased by a blind solver: if a model can answer without the video, a critic names the leak (tone bias, process of elimination) and a reviser rewrites stem or distractors until blind accuracy nears chance or the iteration budget runs out.
Humans check twice. Forty native speakers reviewed 1,500 candidates, flagged 178 as unanswerable, repaired 161, and dropped 17. Twenty-eight more reviewers saw the refined set: 949 accepted, 532 edited, two removed. Sixty-eight people in total.
Closed models use native video APIs at 0.25 FPS, about 3,600 frames for a four-hour tape. Open models sample as densely as their context allows, 64 or 128 frames; Qwen3-VL-8B is also reported at 0.25 FPS.
| Model | Sampling | Narrative | Cultural | Overall |
| Gemini-3-Flash | 0.25 FPS | 84.2 | 68.2 | 76.2 |
| Gemini-3-Pro | 0.25 FPS | 74.1 | 66.0 | 70.0 |
| Gemini-2.5-Flash | 0.25 FPS | 55.8 | 46.9 | 51.4 |
| Qwen3.5-9B | 128 frames | 38.1 | 41.4 | 39.8 |
| Qwen3-VL-8B | 0.25 FPS | 39.5 | 35.4 | 37.4 |
| InternVL3.5 | 64 frames | 28.3 | 34.6 | 31.5 |
| Qwen2.5-VL-7B | 128 frames | 31.1 | 28.2 | 29.7 |
| MiniCPM-o-2.6 | 128 frames | 30.8 | 28.3 | 29.6 |
Chance is 25%. Gemini is about 11 points better on narrative than on culture; open models are similarly weak on both. N.2 sequential flow is the easiest multiple-choice slot. Weak models fail N.1 entity continuity, InternVL3.5 at 20.0%, below chance. Strong models fail N.4 thematic abstraction. The hardest cultural slot is C.3 subtext: Flash scores 57.4% there against a 76.2% overall, the one slot where Pro beats Flash.
On a fixed 500-question subset, raising frames from 8 to 128 moves Gemini-3-Pro from about 64% to 71%, Flash from 53% to 64%, and 2.5-Flash from 39% to 51%. Among open models, Qwen3-VL-8B goes from about 31% to 38%; the rest stay near 30% at 128 frames. Narrative gains 1.4 to 20.5 points with more frames; culture ranges from a 3.9-point drop to a 10.0-point gain. Missing frames explain missed events. They do not explain failed pragmatic reads. Under a matched frame budget Pro leads Flash; on the full set at 0.25 FPS Flash leads, 76.2 versus 70.0. Pro is stronger when evidence is scarce. Flash scales harder once the tape is densely sampled.
Strip the choices and score free-form answers with GPT-5.5 using FActScore atomic-fact recall. Gemini sits at 0.66 to 0.78, open models at 0.21 to 0.56. N.2, the strongest narrative slot in multiple choice, becomes the weakest narrative slot for seven of eight models once the options no longer order the timeline. The narrative-versus-culture ranking also flips: 78.9% of cultural answers recover at least one reference fact, against 66.3% of narrative answers, because cultural stems often sketch the interaction before asking what it meant.
Long-video leaderboards should not stop at "the model found the object at minute 37." NARU ties story continuity and socially implicit meaning to the same hours-long Japanese videos. Open 7B to 9B models are close to unusable here; entity continuity can fall below chance.
Qwen3-VL-8B at 0.25 FPS still scores 37.4%, 14 points behind Gemini-2.5-Flash at the same sampling rate. The gap is not only frame budget. For teams working on Japanese or East Asian media, the 1,481 items are a usable test bed. For English short-form products the absolute numbers will not transfer. The format warning will: multiple choice can fake sequential skill that open-ended generation does not have.
Questions were written by Gemini 2.5 Pro, and the eval set includes Gemini-2.5-Flash and Gemini-3. Native-speaker review can catch invented facts. It cannot catch a question style that favors the Gemini family. Default visual density is still unmatched between open models and Gemini, so part of Table II is frames seen. YouTube sources will rot with takedowns. Diversity sampling used titles and descriptions, not pixels. The cultural slots are tied to Japanese high-context talk; C.2 and C.3 would change flavor in a low-context language. The open-ended judge is GPT-5.5. Three annotators agreed with 96% of 50 majority votes, and it is still a model grading a model.