Omnia de Ego Tempo : Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, Federico Tombari · 2025
Read Omnia de Ego Tempo : Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos by Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, Federico Tombari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13646.txt.
Introduction
Chiara Plizzari and co-authors build a tougher test of time sense for video models in this 2025 paper. First-person video is a stream of close-up hand and object action, where order matters. The team found current question sets leak: models score well from the words alone or one frozen frame. Their new set, EgoTempo, asks only questions that need the whole video watched in sequence.
Ten temporal skills are probed, and cheating by shortcut does not work by design. Leading multimodal models, strong elsewhere, fell well short on the new set. The gap shows how thin present time reasoning is in the egocentric domain. The authors release data and code, hoping the benchmark pushes models that truly track unfolding action.
Catalog record
| CREATOR | Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, Federico Tombari |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Omnia de Ego Tempo : Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, Federico Tombari Abstract Understanding fine-grained temporal dynamics is crucial in egocentric videos, where continuous streams capture frequent, close-up interactions with objects. In this work, we bring to light that current egocentric video question-answering datasets often include questions that can be answered using only few frames or commonsense reasoning, without being necessarily grounded in the actual video. Our analysis shows that state-of-the-art Multi-Modal Large Language Models (MLLMs) on these benchmarks achieve remarkably high performance using just text or a single frame as input. To address these limitations, we introduce EgoTempo , a dataset specifically designed to evaluate temporal understanding in the egocentric domain. EgoTempo emphasizes tasks that require integrating information across the entire video, ensuring that models would need to rely on temporal patterns rather than static cues or pre-existing knowledge. Extensive experiments on EgoTempo show that current MLLMs still fall short in temporal reasoning on egocentric videos, and thus we hope EgoTempo will catalyze new research in the field and inspire models that better capture the complexity of temporal dynamics. Dataset and code are available at https://github.com/google-research-datasets/egotempo.git . Figure 1 : EgoTempo vs existing egocentric Video Question Answering (VideoQA). Left: previous datasets feature tasks solvable with single-frame understanding or commonsense reasoning, while EgoTempo introduces more complex temporal tasks. Right: current Multi-Modal LLMs (MLLMs), such as Gemini [ 35 ] , achieve high accuracy on existing VideoQA datasets using a single frame (51% on EgoSchema [ 26 ] and 41% on EgoTaskQA [ 15 ] ), but only 9% on EgoTempo. On multiple frames, accuracy on other datasets improves by 1.4 \times single-frame performance, while on EgoTempo it improves by 4.3 \times , underscoring its focus on video understanding. 1 Introduction Understanding temporal dynamics in videos is a long-standing and essential topic within computer vision, particularly as the field moves toward more sophisticated models capable of interpreting and reasoning over time. Multi-Modal Large Language Models (MLLMs), spanning both proprietary solutions [ 30 , 35 ] and open-source frameworks [ 18 , 44 , 37 ] , have made impressive advances in video-language tasks. Within this domain, Video Question Answering (VideoQA) stands out as a task that explicitly requires both visual and temporal comprehension, making it a valuable benchmark for evaluating MLLMs’ ability to understand complex, time-based video content. Recent advancements in MLLMs have focused on enhancing video understanding by increasing the volume of data to be processed, such as expanding the number of frames fed into the model. Modern MLLMs now have the capability to process entire video sequences within their expanded context windows [ 35 ] . However, this leads to an important question: is simply providing access to full video data sufficient for achieving temporal understanding? In other words, even with complete observation of the entire video, are models able to reason over the temporal events and relationships? In this realm, egocentric videos introduce significantly more complex challenges. Unlike exocentric videos, which often feature detached observations, egocentric data centers on close-up, frequent interactions with objects, producing a continuous flow of activities that require advanced temporal reasoning. This raises another question: how good are the temporal understanding capabilities of MLLMs when applied to egocentric data? Unfortunately, this area remains relatively under-explored, as the majority of available data predominantly consists of exocentric perspectives [ 33 , 35 ] . Moreover, we highlight critical limitations of the few existing egocentric VideoQA datasets [ 26 , 15 ] due to their tendency to allow successful question-answering based on either commonsense knowledge or isolated frames, without requiring comprehensive temporal reasoning across the video. For example, it is possible to answer the question “What is the main purpose of using aluminum foil?” by just looking at a single frame (Figure 1 , top left). Or, you can guess the answer to “What is the status of the microwave before the person gets something from it?” using commonsense reasoning (see Figure 1 , bottom left). More specifically, our experiments show that Gemini 1.5 Pro [ 35 ] achieves 51% and 41% accuracy on egocentric VideoQA benchmarks EgoSchema [ 26 ] and EgoTaskQA [ 15 ] respectively, using only a single frame (the central one) as input (Figure 1 ). In other words, half of the questions in these datasets can be answered without even looking at the video. This gap highlights the need for a more robust dataset that truly challenges MLLMs’ capacity for temporal understanding, particularly within the context of egocentric interactions. To address this gap, we introduce EgoTempo , a free-form VideoQA dataset specifically designed to assess temporal understanding in egocentric VideoQA tasks across entire video sequences of variable duration ( 45s on average). Our dataset construction pipeline combines a sophisticated automated pipeline followed by rigorous manual review. As a result, EgoTempo features a carefully curated set of 10 different tasks to unlock key insights on different aspects of temporal reasoning of current MLLMs. Some examples from EgoTempo are shown in Figure 1 (middle). Critically, performance on EgoTempo using single frames is limited to just 9% (Figure 1 , right). When provided with frames sampled at 1FPS, performance on existing dataset improve only by up to 1.4 \times over single-frame accuracy, suggesting that current benchmarks may not adequately challenge the temporal reasoning capabilities of these models. In contrast, using the same sampling rate on EgoTempo results in a 4 \times improvement over single-frame performance (Figure 1 ). Moreover, EgoTempo is significantly harder than existing benchmarks, with models achieving only 39% accuracy even when analyzing frames at 1FPS. Indeed, a key insight from our analysis on EgoTempo is that current MLLMs still fall short in effective temporal reasoning on egocentric videos. Simply having access to the full video is not enough—true temporal understanding demands more than just increased frame access . We believe EgoTempo will serve as a catalyst for the development and evaluation of MLLMs with stronger temporal understanding, particularly in the challenging domain of egocentric video data. In summary, our contributions are: • We introduce EgoTempo, an egocentric VideoQA dataset specifically designed to push the boundaries of temporal reasoning. EgoTempo presents 10 diverse tasks, each requiring temporal understanding to capture the complexity of egocentric fine-grained interactions; • We conduct a comprehensive benchmark of EgoTempo using state-of-the-art MLLMs, evaluating their performance with varying amounts of frames in input; • Through extensive analysis across 13 MLLMs, both open- and closed-source, we show that current MLLMs fall short of human-level temporal reasoning, emphasizing the critical need for more sophisticated models capable of handling the complexities of temporal dynamics in egocentric video tasks. 2 Related Works Large Multi-Modal Models. Large Language Models (LLMs) such as ChatGPT [ 27 ] , GPT-4 [ 28 ] , and LLaMA [ 36 ] have demonstrated impressive reasoning and text-based generalization capabilities. The recent development of models capable of integrating visual data, such as GPT-4V(ision) [ 29 ] and Gemini [ 35 ] , represents a significant step forward, extending the scope of LLMs to process both textual and image inputs. Since then, several image-based Multi-Modal Large Language Models (MLLMs) have emerged [ 19 , 1 , 31 ,