EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT

Baoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Yu Qiao, Jiangmiao Pang · 2025

Read EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT by Baoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Yu Qiao, Jiangmiao Pang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23569.txt.

Introduction

Baoqi Pei and seven co-authors built EgoThinker, a framework that teaches AI models to reason from first-person video. In egocentric footage, the person filming stays unseen while their hands reshape the scene, so a model must infer hidden intentions and catch fine hand-object contact. Today's multimodal models handle visible events well but lack that embodied, first-person grasp.

The team first built EgoRe-5M, a question-answering dataset drawn from thirteen million varied first-person clips, with step-by-step reasoning notes and dense labels for hands and objects. They fine-tuned models on that data, then applied reinforcement fine-tuning to sharpen where and when things happen. EgoThinker beat prior methods across several first-person benchmarks and improved sharply on fine-grained localization tasks. The authors released the full code and data for others to use.

Catalog record

CREATORBaoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Yu Qiao, Jiangmiao Pang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23569

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT Baoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Yu Qiao, Jiangmiao Pang Abstract Egocentric video reasoning centers on an unobservable agent behind the camera who dynamically shapes the environment, requiring inference of hidden intentions and recognition of fine-grained interactions. This core challenge limits current multimodal large language models (MLLMs), which excel at visible event reasoning but lack embodied, first-person understanding. To bridge this gap, we introduce EgoThinker , a novel framework that endows MLLMs with robust egocentric reasoning capabilities through spatio-temporal chain-of-thought supervision and a two-stage learning curriculum. First, we introduce EgoRe-5M , a large-scale egocentric QA dataset constructed from 13M diverse egocentric video clips. This dataset features multi-minute segments annotated with detailed CoT rationales and dense hand–object grounding. Second, we employ SFT on EgoRe-5M to instill reasoning skills, followed by reinforcement fine-tuning (RFT) to further enhance spatio-temporal localization. Experimental results show that EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained spatio-temporal localization tasks. Full code and data are released at https://github.com/InternRobotics/EgoThinker . 1 Introduction Humans possess a remarkable ability to reason, plan, and execute complex, goal-oriented behaviors within dynamic real-world environments. Recent works in Multimodal Large Language Models (MLLMs) have advanced the field of visual understanding [ 55 , 1 , 76 , 54 , 12 ] . Techniques such as chain-of-thought prompting [ 74 , 88 ] and reinforcement fine-tuning [ 39 , 27 ] further underscore the potential of MLLMs in high-level reasoning. However, existing approaches mainly address visual reasoning from an observer-centric, third-person viewpoint. This perspective fails to capture the embodied cognitive processes central to human reasoning, which naturally occur from the egocentric perspective. Figure 1 : Overview of our EgoThinker. Unlike general video reasoning, egocentric video reasoning poses unique challenges because it must infer an unobservable camera wearer’s interactions and intentions. EgoThinker addresses this by curating EgoRe-5M, a large-scale egocentric reasoning dataset, and applying a two-stage supervised and reinforcement fine-tuning paradigm. This design empowers robust egocentric reasoning chat, hand–object grounding, and temporal grounding, making EgoThinker a promising foundation for wearable assistants and embodied AI. Egocentric reasoning differs fundamentally from conventional visual reasoning due to the presence of the observer as an active participant in the scene. Thus, models must infer not only visible events but also the internal cognitive states, intentions, and future behaviors of the individual behind the camera. This reasoning process poses several unique challenges: (1) Reasoning for complex tasks: Understanding the rationale behind an action and predicting what comes next demands explicit cause-effect chains rather than isolated event recognition. (2) Human–object interaction recognition: Successful reasoning hinges on accurately localizing hands and manipulated objects. (3) Multi-horizon temporal integration: Egocentric video streams span minutes to hours, requiring models to track evolving context and retain fine-grained details across thousands of frames. Existing egocentric datasets [ 97 , 102 ] are mainly derived from Ego4D [ 25 ] and EgoExo4D [ 26 ] . They provide extensive collections of egocentric videos but lack explicit reasoning chains, temporally spanned annotations, and detailed fine-grained grounding data. Consequently, existing MLLMs [ 84 , 101 , 3 ] , although successful in general visual understanding tasks, often struggle when reasoning about complex tasks in long-term egocentric videos. In this work, we propose EgoThinker , a novel framework designed to enable robust egocentric reasoning in MLLMs. As illustrated in Figure 1 , to overcome dataset limitations, we develop a pipeline to extract egocentric videos from large-scale web data to capture diverse real-world scenarios. Leveraging this, we construct EgoRe-5M, a large-scale egocentric QA dataset, featuring diverse questions spanning from seconds to several minutes. In the dataset, we incorporate detailed Chain-of-Thought (CoT) annotations to explicitly model the causal relationships underlying complex human activities, enabling models to emulate human-like causal inference and planning. Furthermore, recognizing the critical role of hand-object interaction in egocentric reasoning, we additionally introduce specialized data focusing on fine-grained spatio-temporal hand-object interactions. Inspired by recent advances in reinforcement fine-tuning (RFT) [ 39 , 27 ] , EgoThinker employs a two-stage training strategy. Initially, the model undergoes supervised fine-tuning (SFT) [ 84 , 101 ] , utilizing EgoRe-5M to establish foundational understanding and reasoning. Subsequently, we employ RFT using spatio-temporal grounding data via GRPO [ 27 ] method. This paradigm significantly enhances the model’s capability in fine-grained localization, temporal reasoning, and causal inference. Experiments demonstrate that EgoThinker outperforms existing state-of-the-art methods across multiple egocentric benchmarks, showcasing strong performance gains in egocentric QA [ 42 , 21 , 56 ] , long-term video reasoning [ 15 , 63 ] , and spatio-temporal hand-object interaction localization [ 36 , 19 ] . In summary, our contributions are: (1) EgoRe - 5M, a large-scale egocentric QA dataset with chain - of - thought and hand - object annotations curated from diverse video sources. (2) A two - stage training regime combining supervised fine - tuning and reinforcement fine - tuning via GRPO to effectively couple high - level reasoning with low - level grounding. (3) EgoThinker, an MLLM setting new state-of-the-art on multiple egocentric video benchmarks, demonstrating coherent egocentric reasoning and precise spatial and temporal grounding. 2 Related work Egocentric Video Understanding. Egocentric video understanding has recently garnered increasing research attention, with the introduction of large-scale egocentric datasets such as Ego4D. Previous works mainly focus on action understanding [ 71 , 33 , 24 , 32 , 34 , 68 , 29 , 10 ] , vision-language pretraining [ 56 , 35 , 108 , 69 , 92 ] and hand-object interaction understanding [ 100 , 9 , 22 , 93 ] . Recently, some methods [ 5 , 97 , 102 , 30 , 37 ] have attempted to use MLLMs for egocentric video understanding, and some targeting long-form egocentric videos [ 94 , 86 ] . Different from these works, EgoThinker is the first model that enables reasoning and precise hand–object understanding in first-person videos. Multimodal Large Language Models. Recent advancements in MLLMs [ 55 , 1 , 76 , 31 , 54 , 12 , 6 ] have shown robust comprehension and perception abilities. Video-LLaVA [ 55 ] and Videochat2 [ 51 ] enable MLLMs with general video understanding and temporal localization capability. Recent works extend to diverse domains including long-form video understanding [ 104 , 11 , 82 , 44 ] , robot learning [ 89 , 45 ] , and spatio-temporal perception [ 66 , 61 ] . Chain-of-Thought (CoT) prompting [ 88 ] has proven effective in eliciting multi-step reasoning in LLMs. Recently, some works [ 13 , 74 , 14 ] use CoT techniques to enhance MLLM’s visual reasoning capabilities. In our work, we construct large-scale QA samples with causal CoT captions to equip MLLMs with egocentric reasoning ability. Reinforcement Learning for MLLMs. Recently, OpenAI-o1 [ 39 ] and DeepSeek-R1 [ 27 ] have shown that reward-driven fine-tuning can substantially enhance LLM reasoning. For MLLMs, many works [ 110 , 58 , 99 , 20 , 70 , 59 , 95 , 103 , 60 ] focused on leveraging reinforcement learning (RL) techniques with verifiable rewards to enhance visual reasoning capabilities with only a small amount of data. Videochat-R1 [ 52

Read the complete text