EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos
Shulin Tian, Junsu Kim, Shuai Liu, Hao Li, Yujiao Shen, Sihan Li, Zhe Yang, Yeongon Kim, Feiyu Li, Jialin Wu, Yichi Zhang, Wenhui Wang, Runmao Yao, Yuhao Dong, Zhaoxi Chen, Fangzhou Hong, Antonino Furnari, Jingkang Yang, Hongyuan Zhu, Ziwei Liu
cs.CV
2026-09-30
A 100-hour egocentric tool-use corpus and 1,000-question benchmark: Gemini-3.1-Pro reaches 66.9% vs 83.2% for humans; fine-tuned Qwen3-VL-8B climbs from 50.0% to 60.9%.
Most real-world embodied tasks are tool-mediated: picking the right wrench, noticing that scissors will not handle tough asparagus stems and switching to a knife. Understanding such activities means tracking hand-tool-object geometry, procedural progress, and the causal effect of each action on the target object. Video multimodal models score well on captioning and general video QA but struggle with this kind of reasoning.
Both sides are stuck. Mainstream egocentric datasets (EPIC-KITCHENS, Ego4D, Assembly101, HOI4D) are organized around activities, objects, or hand-object interactions; annotations stop at "cook eggs" while the spatula and pan stay invisible. No existing benchmark isolates tool-mediated reasoning either. EgoTools fills both gaps with a 100-hour densely annotated corpus, a 1,000-question four-track benchmark, and a 184,679-example instruction-tuning set.
EgoTools-Data was recorded with HOMIE, a head-mounted rig with four fisheye cameras plus audio and motion signals; the rectified front-left stream is the canonical view at 1024x1024, 20 FPS, spanning seven domains from kitchen and household to research lab and repair workshop. Two collection modes reflect a deliberate trade-off: structured task-guided sessions give clean boundaries and reliable annotation, while open-ended sessions hand participants only a broad topic, so spontaneous tool substitution, retries, and error recovery emerge naturally. That second mode is where tool-causal narration comes from.
Annotation runs on three tracks:
The training pool and the benchmark pool share zero source videos (the benchmark keeps 40.34 hours). The training pool yields 184,679 instruction examples, 37.8% of which directly supervise the four abilities the benchmark tests. EgoTools-Bench has 1,000 8-way multiple-choice questions (chance 12.5%) averaging 4.19 minutes per question: Affordance & Causality 363, Perception & Grounding 236, Procedural Dynamics 222, Spatial Reasoning 179. 900 are human-written; 100 spatial questions are generated from 3D annotations and human-verified. A reliability study on 250 questions with two blind judgments each found 89.8% agreement with the answer key, and 94.4% of items were confirmed to have a unique best answer.
| Model | EgoSchema | EgoTools-Bench | Perception & Grounding |
| Human expert | – | 83.2 | 85.2 |
| Gemini-3.1-Pro | – | 66.9 | 51.7 |
| GLM4.1V-Thinking | 66.1 | 50.7 | 50.0 |
| Qwen3-VL-8B-Instruct | 69.0 | 50.0 | 45.9 |
| EgoTools-8B (fine-tuned) | 67.6 | 60.9 | 59.7 |
Gemini-3.1-Pro tops the board at 66.9% overall but manages only 51.7% on Perception & Grounding, 18 to 23 points below its own scores on the other three tracks. Open-source instruct models sit between 38.9% and 50.0%, more than 30 points behind the human expert. The same Qwen3-VL-8B-Instruct that scores 69.0% on EgoSchema gets 50.0% here: broad egocentric activity understanding does not carry tool-use reasoning.
Fine-tuning works but is lopsided. A single-stage full SFT of the language component (vision encoder frozen) lifts Qwen3-VL-8B-Instruct from 50.0% to 60.9%: +9.3 Affordance & Causality, +13.8 Perception & Grounding, +11.0 Procedural Dynamics, and -11.0 Spatial Reasoning. EgoSchema slips slightly from 69.0 to 67.6.
Input ablations on one backbone (frame-list pipeline, not directly comparable to the main MP4 protocol): ordered 64 frames 51.7%, the same frames shuffled 49.3%, middle frame only 46.9%, text-only 42.7%. Text-only sitting 30 points above the 12.5% chance line means residual language priors survive the anti-shortcut filtering. Perception & Grounding depends most on visual evidence (+13.9 over text-only), Procedural Dynamics is most order-sensitive (-3.2 shuffled), and Spatial Reasoning mainly needs multi-frame coverage (-11.9 single frame) while caring little about order (-1.1 shuffled).
Reporting four tracks separately makes the benchmark diagnostic: it tells you whether a model fails at visual grounding or at procedural and causal reasoning. Perception & Grounding is the shared weak spot, with even Gemini at 51.7%; "which exact tool touches which object" remains a hard bottleneck for video models.
For data builders, 184,679 examples buying 10.9 points under strict source-video separation shows the annotations are actionable supervision, and both the dataset and the fine-tuned 8B model are public on HuggingFace, ready for continued training or mixing. The boundary is equally clear: spatial reasoning drops after fine-tuning, so this recipe does not fix that track.
The paper concedes several points: the question pool is long-tailed, mostly kitchen videos, so the primary analysis stays at track level; human evaluators could replay videos freely while models got a fixed 64-frame budget; transfer to EgoPlan is negative (42.3 down to 35.2), and the claim is explicitly narrowed to tool-centric reasoning.
Open questions from reading it: text-only accuracy of 42.7% is high for a benchmark built around visual evidence; all 100 hours come from students and researchers at Asian university sites, so professional depth is limited; the 11-point spatial regression gets no attribution experiment; and with 69,554 episode-level multiple-choice items in the training mix sharing the benchmark's answer format, how much of the +10.9 is capability versus format adaptation is never separated.