TennisVAR: A Stroke-Evidence-Grounded Multimodal Large Language Model for Tactical Reasoning in Tennis Videos
Yifan Mei, Qingling Shi, Changli Wu, Jiayuan Rao, Jiayi Ji, Liujuan Cao
cs.CV
2026-08-13
TennisVAR parses rally strokes then reasons over a stroke graph. On TRACE it reaches 73.04 T-F1@8 and 56.19 T-IoU@4, 19.94 and 33.03 points above the strongest SFT baselines.
Tennis video research has run on two tracks that barely meet. TrackNet and F3Set can pin racket-ball contacts to a frame and label who hit, which stroke, which direction. They treat each stroke as an independent event, so they cannot explain how three earlier balls forced a later approach. TennisTV, TennisExpert, and general multimodal LLMs can write commentary and answer tactical questions, but the justification often comes from the rally outcome or from language priors. You cannot check which strokes actually support the claim.
This paper turns that gap into a rally-level task: stroke-evidence-grounded tactical reasoning. Given a rally clip and a natural-language question, the model must jointly produce an open-ended answer, a three-level tactic label, an ordered set of supporting strokes, and the decisive key actions inside that set. Every evidence stroke is anchored to its contact frame. Correct wording is not enough. The reasoning chain has to point back to events that happened on court.
TRACE is the benchmark that makes the task measurable. It extends F3Set's densely timestamped tennis events into multi-stroke tactics and evidence-grounded QA. Source video covers men's and women's professional matches from Grand Slams, tour events, the Olympics, and team competitions: 109 matches, 72 players, 11,189 rallies, 41,485 strokes, 25,429 expert-verified tactical units, and 11,189 questions. Splits are cut at the source-match level (7,119 / 1,805 / 2,265) so the same match never leaks across sets. The tactic ontology has 6 / 17 / 25 classes covering serve, return, baseline construction, net transition, and defensive counterattack, plus a no-primary-tactic class. Questions come in three tiers: 3,643 factual (Q1), 6,376 tactical (Q2), 1,170 decision (Q3). A language model proposes candidate tactical units; three tennis experts, calibrated on a shared set, review each unit with at least two people. Multiple model families then draft QA pairs independently. Human annotators rewrite them without knowing which model wrote the draft, and mark the minimal supporting-stroke set and the key-action subset. Player names are replaced with camera-relative [Player Near] and [Player Far].
TennisVAR runs in three stages: event, relation, evidence, tactic.
The Event Parsing Module (EPM) does not dump the raw clip into a VLM. It fuses DINOv3 appearance, short-term motion, and TrackNet ball trajectory, then uses F3Set's F3ED encoder to decode an ordered stroke sequence: contact frame, fused visual feature, hitter, stroke type, direction, outcome. Contact detection uses continuous-target focal BCE. Attribute losses fire only at annotated contacts. Later graph reasoning sits on this sequence. Contact frames are inherited from EPM, not predicted again.
The Tactical Graph-Guided Temporal Reasoner (TGTR) treats each stroke as a node with two edge types: adjacent-stroke temporal edges, and same-player decision edges that skip the opponent's return. Relation-conditioned message passing plus a Transformer yields stroke-level context and a rally vector. Conditioned on the question, two heads score whether a stroke is supporting evidence and whether it is a key action. An Evidence Router weights strokes by the evidence scores and fuses rally, selected strokes, and the question into one vector that all three tactic heads read. Training never uses oracle stroke indices. Annotated evidence frames are matched one-to-one to predicted contacts, so train and test both sit in predicted-event space.
Only then does language generation run. Qwen3-VL-8B stays frozen except for rank-32 LoRA. The generator sees sparse global frames, local windows around selected contacts, and a serialized event table. Tactic, evidence, and key-action fields come from TGTR. The language model verbalizes a structure that already carries evidence. Training used 8 NVIDIA H20 96GB GPUs.
Zero-shot general models write fluent answers and miss the strokes. GPT-5.5 scores 37.03 T-F1@8 and 24.54 T-IoU@4. TennisVAR reaches 73.04 and 56.19.
Against the strongest supervised fine-tunes, the evidence gap is larger than the language gap:
| Method | T-F1@8 | T-IoU@4 | Hier. tactic F1 | Key-action acc. | Total |
| InternVL3-8B SFT | 53.10 | 23.16 | 59.91 | 44.75 | 44.81 |
| Qwen3-VL-8B SFT | 49.46 | 21.06 | 64.49 | 47.13 | 44.83 |
| TennisVAR | 73.04 | 56.19 | 70.98 | 52.27 | 57.11 |
Gains versus the corresponding strongest SFT baseline are 19.94 T-F1@8, 33.03 T-IoU@4, and 6.08 hierarchical tactic F1. Text metrics barely move: versus the same-backbone Qwen3-VL-8B SFT, ROUGE-L / CIDEr / BLEU-4 rise 1.78, 1.80, and 1.83. Lexical overlap does not measure whether the answer sits on the right strokes.
By difficulty, Q1 factual T-F1@8 is 83.50, Q2 tactical 71.08, Q3 decision 54.88, still 14.71 above the strongest SFT. Drop TGTR and Total falls 9.07, T-F1@8 falls 17.14, key-action accuracy falls 10.92. Drop the Evidence Router and hierarchical tactic F1 falls 10.18. Among EPM inputs, removing DINOv3 costs 6.10 Total, the largest overall drop; removing TrackNet or the motion stream costs 10.91 and 8.82 T-F1@8.
For anyone building sports video QA or evidence-grounded video reasoning, this paper separates answers that sound like analysis from answers that can name the supporting events. Zero-shot MLLMs are close to unusable on TRACE evidence metrics. Fine-tuning Qwen3-VL still only reaches 49.46 T-F1@8. The lift comes from parsing strokes into discrete events and routing question-conditioned evidence over a stroke graph.
The usable setting is narrow: professional tennis broadcasts, and tactical QA that has to be auditable. This is not a general video-understanding recipe. Language generation barely touches the frozen backbone, and the text-metric gains are thin. The piece worth copying is the event-graph pipeline, not another commentary model.
The paper has no dedicated limitations section. The numbers already show the holes. Key-action accuracy is 52.27, barely over half. Frame accuracy F-Acc@8 is 47.86, so contact-frame alignment is still coarse. Q3 has only 1,170 items, and T-F1@8 sits 28.62 below Q1; decision-level reasoning is not stable yet.
All video is professional broadcast, built on F3Set timestamps. Amateur play, multi-camera setups, and heavy occlusion are untested. Players are camera-near versus camera-far, so the model never sees named-player tactical habits. Tactical units start as language-model proposals, and questions are model-drafted then rewritten, so the label distribution may inherit generator phrasing. Training took 8 H20 96GB GPUs with staged EPM / TGTR / LoRA, which is not cheap to reproduce. A project page is listed; the fetch record has no GitHub repo.