EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning
Yuqi Tang, Yang Shi, Yuran Wang, Tengfei Liu, Rundong Wang, Zongyang Qiu, Qixun Wang, Zhuoran Zhang, Xuanyu Zhu, Yifan Xu, Yiyan Ji, Shujie Li, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei Wan, Chengwei Qin
cs.CV
2026-11-20
A 9B model trained on 50k coarse-to-fine captions with GRPO scores 78.84 on PhysCapBench, above Gemini 3.1 Pro, and leads open-source models on five egocentric benchmarks.
In first-person video the wearer's body is mostly out of frame, and a head turn swaps the scene. General multimodal models can write at length about whatever is visible, and they still drop actions that last a second or two, swap their order, or treat head motion as the body walking. Dedicated egocentric captioners usually swing the other way: they stay on hands and objects, and leave navigation, conversation, and looking around thin.
The format fights the use case. A dense paragraph has detail and no reliable start or end. A timestamped caption is often one sentence per step, with nowhere to put the instant an object changes state. An assistant that uses captions as memory, and a robot pipeline that cuts clips by timestamp, need the detail and the clock together.
EgoCaptioner writes two levels. The overview sets place, lighting, where people and objects are, and what the wearer is doing across the clip. Steps cut the video into contiguous, non-overlapping events, then into ordered steps with start and end times. A key instant is a sentence that starts with At [t]. Full-video captions, overview-only captions, and event-conditioned captions share that schema. In the last of these, the prompt names the event and the model has to locate it.
Which details appear depends on the event type. Manipulation covers each hand's role, how the object changes, and where it is put down. Locomotion covers how the wearer travels and the route past landmarks, using camera bob to separate walking from riding. Social steps record who is there and how the turns go. Attention steps follow gaze, occlusion, and the moment a target comes back.
Annotation locks the structure before the prose, so one pass is not asked to segment and describe at the same time. Gemini-3.1-Pro writes the overview. Seed2.1 pro marks boundaries, event types, and a step outline. Gemini then sees only that event, plus at most one second of context on each side, and expands the outline. Video longer than 30 seconds is cut into pieces of at most 30 seconds with 0.5 seconds of overlap, then merged. Seed2.1 pro checks the draft on the full video and patches actions, times, and contradictions. Eight egocentric sources produce 50,000 examples, split evenly across the three tasks, with clips up to 120 seconds. When pose or trajectory exists, MotionScript turns the sensors into timed lines about body motion and the head-mounted view. Those lines do not depend on the caption, and they are extra reference facts in the later reward.
The backbone is Qwen3.5-9B, LoRA rank 32, two epochs of supervised fine-tuning at learning rate 1e-4. GRPO then runs on 8,000 prompts with eight sampled captions each. Both stages use 16 GPUs. The fact reward covers motion, task focus, scene and purpose, and key moments. Gemini-3.1-Pro scores each fact from 0 to 1, and the reward is the F1 of precision and recall. A key moment is correct only when the time and the change both match. The time reward greedily matches steps by temporal IoU and divides by the larger of the two step counts, so cuts that are too fine or too coarse lose points. The style reward penalizes manipulation steps that describe the hand and not the effect on the object, and sentences that repeat background already given. Full-video and event tasks use weights 0.6, 0.2, and 0.2 on fact, time, and style. Overviews use 0.9 on facts and 0.1 on format: start with the scene, end with what the wearer is doing. Truncated, looping, or nonpositive-duration outputs score -1. A KL term pulls the policy toward the supervised model.
Five egocentric benchmarks. EgoMe scores text against references. EgoHome PSJ compares predicted steps with human subtask timelines. PhysCapBench Overall averages seven physical-detail axes. Minerva-Ego and EgoTempo are run as caption-to-QA: Gemini 3.8 Flash answers from the caption alone and then grades the answer. Frames are taken at 4 fps, capped at 240.
| Model | PhysCap | PSJ | EgoTempo | Minerva |
| EgoCaptioner | 78.84 | 75.51 | 38.40 | 18.57 |
| Qwen3.5-9B | 51.24 | 46.90 | 21.60 | 0.00 |
| Gemini 3.1 Pro | 75.06 | 52.86 | 23.60 | 4.29 |
| Gemini 3.8 Flash | 82.77 | 71.09 | 41.20 | 7.14 |
| GPT-6-Astra | 84.87 | 79.42 | 52.60 | 15.71 |
Against the 9B backbone, PhysCapBench is up 27.6 points and EgoTempo accuracy is up 16.8 points. EgoCaptioner ranks first among the open-source models on all five benchmarks. Qwen3.8-27B reaches only 69.55 on PhysCapBench. Proprietary models do not fall in a single order. PhysCapBench beats Gemini 3.1 Pro and trails Gemini 3.8 Flash and GPT-6-Astra. PSJ beats Gemini 3.8 Flash and trails GPT-6-Astra. Minerva-Ego beats every proprietary model in the table. Overview BLEU-4 is 4.03 and METEOR is 20.40, the highest figures reported, proprietary models included. Step METEOR is 12.26 and SODA-d is 56.38.
CapRL-Video-4B scores 75.94 on PhysCapBench, about 2.9 points lower, at the same mean length of 548 words. The larger per-axis gaps are physical mechanism and causality, up 5.88, and temporal order, up 4.57.
Ten thousand supervised examples already move PSJ from 46.90 to 67.79. Fifty thousand reach 71.63. METEOR goes the other way at 10k and 25k, 16.06 and 16.37, below the backbone's 17.58, and recovers only at 50k, to 19.91. After GRPO the numbers are METEOR 20.40, SODA-d 56.38, and PSJ 75.51. Remove the fact reward and all three sit below that 50k supervised model. Remove the time reward and PSJ falls to 66.81 while METEOR barely moves. Remove the style reward and both SODA-d and PSJ fall. Coarse-to-fine supervised training already reaches 18.57 on Minerva-Ego, and the GRPO model is still at 18.57. Over that same step, PSJ moves from 71.63 to 75.51 and SODA-d from 53.25 to 56.38.
Same videos, same volume, one-pass Gemini 3.1 Pro versus coarse-to-fine: EgoTempo 26.60 versus 36.40, Minerva-Ego 7.14 versus 18.57, PSJ 62.89 versus 71.63. The one-pass captions have coarser boundaries and fewer step-level facts.
Retrieval memory and robot pretraining need the seconds a step occupies, and how a hand carries an object into the next state. A 9B model plus LoRA is enough to lead the open-source set. The pieces worth copying are the four event styles, structure before detail, and fact and time rewards that do not replace each other. Next to the strongest proprietary models the gap is incremental. On PhysCapBench, Gemini 3.8 Flash and GPT-6-Astra are still higher.
The paper has no limitations section. Gemini-3.1-Pro writes the training captions and judges the fact reward. Gemini 3.8 Flash reads the captions and grades the answers in caption-to-QA. References, rewards, and the eval reader come from one model family, so the scores can favor that house style. Human timelines, the main outside anchor, are on EgoHome.
Pose facts enter training only when sensors exist, and inference still sees frames alone. The introduction treats head motion versus body motion as a standing failure, then never reports the split with and without pose. Step scores also punish format. Tarsier2, AVoCaDO, and EgoGPT land at 0 because they rarely emit a timestamp a parser can read. Ten thousand examples already take most of the PSJ gain while METEOR is still below the backbone. Alignment and overlap with the reference text do not move together. Training stops at 120 seconds, and longer video is squeezed into 240 frames. The four event types are fixed ahead of time. No human preference numbers are reported.