EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs CCS: Computing methodologies Activity recognition and understanding

Yang Dai, Dian Jiao, Tianwei Lin, Wenqiao Zhang · 2026

Read EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs CCS: Computing methodologies Activity recognition and understanding by Yang Dai, Dian Jiao, Tianwei Lin, Wenqiao Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19559.txt.

Introduction

Dai and team introduce EgoCoT-Bench, a benchmark for grounded, verifiable step-by-step reasoning on first-person video. Existing egocentric benchmarks rarely test whether model rationales rest on explicit space-time evidence for fine-grained hand-object operations. The new set holds 3,172 verifiable question-answer pairs over 351 egocentric videos in four task groups, built with a scene-graph pipeline and refined by human annotators. Tests show persistent difficulty with fine-grained egocentric reasoning, and many models give right answers backed by contradicting evidence.

Catalog record

CREATORYang Dai, Dian Jiao, Tianwei Lin, Wenqiao Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30425
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19559

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs CCS: Computing methodologies Activity recognition and understanding Yang Dai, Dian Jiao, Tianwei Lin, Wenqiao Zhang Abstract . The rapid development of Multimodal Large Language Models (MLLMs) has led to growing interest in egocentric video understanding, specifically the ability for MLLMs to recognize fine-grained hand-object interactions, track object state changes over time, and reason about manipulative processes in dynamic environments from a first-person perspective. However, existing egocentric video benchmarks suffer from limited grounded rationale evaluation , offering limited support for fine-grained operation-centric reasoning and rarely examining whether model rationales are grounded in explicit spatio-temporal evidence. To address this gap, we introduce EgoCoT-Bench , a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning with explicit step-by-step rationale annotations. Overall, EgoCoT-Bench comprises 3,172 verifiable QA pairs over 351 egocentric videos separated into four task groups for a total of 12 sub-task groups, encompassing perception and retrospection, anticipation, and high-level reasoning. The benchmark is constructed through a spatio-temporal scene graphs (STSG) guided generation framework and is further refined by human annotators to ensure correctness, egocentric relevance and fine-grained quality. Experimental results show continuing difficulties with egocentric fine-grained reasoning and further reveal that many multimodal models produce explanations that are answer-correct, but have evidence that is inconsistent with the answer. We hope EgoCoT-Bench can serve as a useful testbed for grounded and verifiable reasoning in egocentric video understanding. Project page and supplementary materials are available at: https://dstardust.github.io/EgoCoT/ . Figure 1 . Overview of EgoCoT-Bench. EgoCoT-Bench is a fine-grained benchmark for grounded and verifiable operation-centric reasoning in egocentric videos, containing 3,172 QA pairs over 351 videos across four task groups and 12 subtasks. It is built through an STSG-guided human verification pipeline with explicit spatio-temporal evidence and rationale annotations. 1. Introduction The rapid progress of multimodal large language models (MLLMs) has greatly advanced video understanding, opening up new possibilities for question answering, temporal reasoning, and embodied perception ( 48 ; 27 ; 36 ; 26 ; 49 ; 53 ; 9 ) . Among these directions, egocentric video understanding is of particular importance for real-world assistive agents and embodied systems ( 10 ; 16 ; 28 ) , since first-person observations directly capture how a user manipulates objects, shifts attention, and interacts with the surrounding environment during task execution. Compared with generic third-person videos, egocentric videos require models to reason about ongoing hand-object interactions, local state changes, and short-horizon action evolution from the operator’s own viewpoint ( 35 ; 38 ) . Table 1 . Comparison with representative video and egocentric benchmarks. Benchmark #Clips #Samples Question Type Annotation Egocentric CoT / Rationale Temporality Spatial Grounding Metric Video-MME ( 15 ) 900 2,700 Close Human ✗ ✗ ✗ ✗ Accuracy MMVU ( 41 ) 1,529 3,000 Open/Close Human ✗ ✓ ✓ ✗ Accuracy LongVideoBench ( 19 ) 3,763 6,678 Close Human ✗ ✗ ✓ ✗ Accuracy EgoSchema ( 21 ) 250 hours+ 5,000+ Close Human ✓ ✗ ✓ ✗ Accuracy EgoThink ( 6 ) 595 700 Open Human ✓ ✗ ✗ ✗ LLM-Judge EgoTempo ( 8 ) 365 500 Open Auto&Human ✓ ✗ ✓ ✗ LLM-Judge MultiHop-EgoQA ( 4 ) 360 1,080 Open Auto&Human ✓ ✗ ✓ ✗ Accuracy/LLM-Judge EOC-Bench ( 47 ) 656 3,277 Open/Close Human ✓ ✗ ✓ ✓ Accuracy EASG-Bench ( 34 ) 221 1,807 Open Auto ✓ ✗ ✓ ✓ LLM-Judge EgoCoT-Bench (Ours) 351 3,172 Open/Close Auto&Human ✓ ✓ ✓ ✓ Accuracy/LLM-Judge However, understanding dynamic object interactions in egocentric videos remains particularly challenging. Owing to the first-person viewpoint, manipulated objects are often only partially visible, intermittently leave and re-enter the field of view, and are frequently occluded by the wearer’s hands under rapid camera motion. The problem is further compounded by cluttered scenes and the presence of visually similar objects, which make the correct interaction target difficult to identify from instantaneous appearance alone ( 50 ; 52 ; 51 ) . More fundamentally, answering egocentric questions requires reasoning over temporally evolving evidence rather than relying solely on the current frame, including prior contact history, earlier object states, and the immediate context of an ongoing manipulation sequence ( 35 ; 38 ; 13 ) . Despite growing interest in egocentric understanding, existing benchmarks still suffer from limited grounded rationale evaluation . As summarized in Table 1 , general video benchmarks such as Video-MME ( 15 ) , MMVU ( 41 ) , and LongVideoBench ( 19 ) have substantially advanced video QA and temporal reasoning, but they are not designed for first-person interaction understanding and provide limited support for spatial grounding. Egocentric benchmarks such as EgoSchema ( 21 ) , EgoThink ( 6 ) , EgoTempo ( 8 ) , and MultiHop-EgoQA ( 4 ) move evaluation closer to first-person settings, especially for temporal or open-ended reasoning, but they still provide limited support for explicit rationale supervision and fine-grained spatial grounding. More recent benchmarks such as EOC-Bench ( 47 ) and EASG-Bench ( 34 ) further incorporate egocentric temporal and spatial evaluation, but they still offer limited support for jointly assessing rationale faithfulness, temporal sensitivity, and evidence-aware evaluation. (a) Dimensions of EgoCoT-Bench (b) Video source distribution Figure 2 . Overall statistics of EgoCoT-Bench. Top: representative video sources in the benchmark. (a) Dimensions of EgoCoT-Bench. (b) Distribution of EgoCoT-Bench samples. To address this gap, we introduce EgoCoT-Bench , a fine-grained egocentric benchmark for grounded and verifiable operation-centric reasoning with explicit step-by-step rationale annotations and spatio-temporal grounding. EgoCoT-Bench contains 3,172 QA pairs over 351 egocentric videos and is organized into four task groups with 12 fine-grained subtasks. These tasks cover egocentric grounding and perception, spatio-temporal retrospection, predictive and causal inference, and high-level grounded reasoning, targeting key capabilities required for first-person manipulation understanding beyond generic scene comprehension. A central design goal of EgoCoT-Bench is to evaluate not only answer correctness but whether MLLMs reasoning are grounded in explicit first-person evidence. To this end, we construct the benchmark using a spatio-temporal scene graphs (STSG)-guided generation framework. Candidate QA samples are first derived from structured egocentric interaction traces, and subsequently refined through human annotation to ensure semantic correctness, first-person relevance, and fine-grained reasoning quality. Each accepted sample is further augmented with structured evidence annotations—including timestamps, object identities, interaction relations, action history, and localized bounding boxes-enabling evaluation at both the answer and the evidence grounding level. Using EgoCoT-Bench, we benchmark a range of representative MLLMs such as GPT ( 29 ; 30 ) , Qwen ( 2 ; 32 ) and LLaVA ( 23 ; 1 ) series, and observe that fine-grained egocentric reasoning remains highly challenging. While many models can produce correct answers, their underlying rationales are often temporally incomplete, weakly grounded, or inconsistent with the available object-level spatio-temporal evidence. This reveals a notable gap between answer correctness and reasoning faithfulness in current models, which may in turn limit performance gains and lead to error accumulation in more complex scenarios. Our findings highlight the importance of moving beyond final answer accuracy, advocating instead for evaluation protocols that explicitly assess whether model reasoning is consistent with the underlying spatio-temporal evidence in egocentric video understanding. In summary, our contributions are three-fold: (1) we introduce EgoCoT-Bench , a fine-grained egocentric benchmark for operation-centric reasoning, comprising 3,172 QA pairs over

Read the complete text