Kling's EgoCaptioner uses 50k samples and GRPO to master egocentric video captioning
KlingTeam · hf · 2026-11-20
KlingAI Research with HKUST(GZ), PKU, and Tsinghua present EgoCaptioner, targeting MLLM failures on egocentric video: missed brief actions, confused ordering, and misread wearer motion.
- Output pairs a video-level overview (setting, participants, overall activity) with timestamped steps whose narration adapts to perceptual demand and marks key moments
- A coarse-to-fine annotation pipeline builds event structure before expanding step descriptions, yielding 50,000 training examples from eight egocentric sources; pose-derived motion facts supplement factual supervision when sensors are available
- Group Relative Policy Optimization (GRPO) applies complementary rewards for factual accuracy/coverage and step-level temporal alignment, penalizing fragmented narration
- Results: leading open-source performance across five egocentric benchmarks; surpasses Gemini 3.1 Pro on PhysCapBench and all evaluated proprietary baselines on Minerva-Ego caption-to-QA. Ablations confirm the value of factual/temporal rewards and coarse-to-fine annotation
More from Multimodal
- Alibaba's TaoTex diffusion model generates native 3D materials, beating TRELLIS.2 — jiqizhixin · 2026-11-20
- AI turns Mediterranean illustration into interactive 3D world; Three.js wall-crash demo costs $6.98 — Saboo_Shubham_ · 2026-11-20
- Kling team's Coda-DaVinci reimagines video generation via Code2Video agents — KlingTeam · 2026-11-20
- Tencent's WeMM-Video pushes open-source omni-modal video understanding — tencent · 2026-11-20
- WM^MLLM: Princeton and NVIDIA turn multimodal LLMs into editable world models — OpenEnvisionLab · 2026-11-20
- Open-source repo offers 500+ AI-generated low-poly 3D landmarks for maps under CC0 — tomjohndesign · 2026-11-20