Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition
Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh · 2025
Read Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition by Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04121.txt.
Introduction
Julia Lee Romero and co-authors recognize action steps in first-person video in a 2025 paper. Head-mounted cameras give shaky, blocked, fast-changing views, which makes naming each step of a task hard. Their method turns a video into a graph: each clip becomes a node, and links join related clips. Steps are then read off as node labels.
During training, clips from outside camera views join the graph as extra nodes, teaching the system views it will not have at test time. Several link designs were tried. On a large public dataset of paired inside and outside views, the graph method beat earlier approaches by over twelve points of accuracy. The graphs stay sparse, so the method runs cheaply. Extra signals like narration and depth were also weighed for their help.
Catalog record
| CREATOR | Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-04121
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh Abstract Egocentric videos capture scenes from a wearer’s viewpoint, resulting in dynamic backgrounds, frequent motion, and occlusions, posing challenges to accurate keystep recognition. We propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos, and leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos. Our approach consists of constructing a graph where each video clip of the egocentric video corresponds to a node. During training, we consider each clip of each exocentric video (if available) as additional nodes. We examine several strategies to define connections across these nodes and pose keystep recognition as a node classification task on the constructed graphs. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods by more than 12 12 points in accuracy. Furthermore, the constructed graphs are sparse and compute efficient. We also present a study examining on harnessing several multimodal features, including narrations, depth, and object class labels, on a heterogeneous graph and discuss their corresponding contribution to the keystep recognition performance. 1 Introduction Analyzing skilled single-person activities is emerging as an enabler for next-generation applications such as virtual assistants and human-AI collaborations Holoassist_Wang_2023_ICCV ; assembly101_sener2022 . Among many research directions, keystep recognition has been instrumental in understanding task structure and procedural analysis from single-person activities. Keystep recognition as a task has been benchmarked on either first-person (egocentric) videos or third-person (exocentric) videos due to the absence of paired ego-exo view videos grauman2022ego4d . Even when paired ego-exo views exist, in many instances, the exo view is not available at inference time, as is the case for wearable devices. However, egocentric videos present many challenges for keystep recognition due to frequent movement of the viewport, changing background, and dynamic objects that move in and out of the frame. Therefore, the community has started to see renewed interest in developing keystep recognition methods that can leverage multi-view data only during training for improved performance at inference time, resulting in a collaborative effort to create a benchmark dataset egoexo4d . We develop a compute-efficient approach for fine-grained keystep recognition on procedural egocentric videos that leverages long-term dependencies more efficiently for improved egocentric-only predictive performance while allowing for further improvement with a variable number of exocentric videos available only during training as demonstrated in Figure 1 . (a) Egocentric Vision Graph (b) Multi-view Vision Graph Figure 1 : Graph-based representation learning for keystep recognition. Learning leverages additional exocentric views via multi-view alignment. Inference is always over the egocentric view only. In this work, we focus on the the newly released Ego-Exo4D dataset egoexo4d and the associated benchmark task challenge of fine-grained keystep recognition, which aims to improve video understanding in egocentric videos. While the task is to classify temporal action segments, or keysteps , in egocentric videos, multiple exocentric video takes are available during training. This dataset is complex and diverse with videos capturing procedural actions relating to cooking, bike repair, and health in different settings and with large variations in video lengths. The keystep actions are fine-grained and many involve manipulation of the same objects in the scene. Additionally, keysteps may occur over different time scales, rendering keystep recognition a challenging task. Fine-grained keystep recognition, which is recognizing the step a camera wearer is performing, is non-trivial and warrants addressing the inherent unique challenges associated with this task. First, keysteps in the same activity may look similar and may involve differentiating subtle differences in hand-object interactions, often in the presence of occlusions and head motion. Second, this task should leverage long-form video reasoning since the sequences of keysteps and the overall activity are correlated. Finally, this task encourages investigation into methods that can effectively learn from multi-view video data and infer from the egocentric video only to be applicable for real-world use. To summarize, below is the list of our contributions. • Graph-based representation learning for long-videos: We propose a compute-efficient, graph-based representation learning framework for modeling temporal dynamics in long-form videos. We refer to this framework as MAGLEV, Multi-view Alignment with Graph for Long and Effective Video understanding. • Multi-view alignment with graph: MAGLEV fuses complimentary information from variable number of multiple views in a flexible manner by learning multi-view alignment. • Heterogenous graph learning using multimodal alignment: We also present a study on how to leverage complimentary multimodal information using the MAGLEV-Hetero framework. We consider additional modalities such as detected object classes, narrations, and depth maps. • Extensive experiments on Ego-Exo4D: MAGLEV notably outperforms existing egocentric methods by more than 15\% , and it can leverage multi-view (egocentric and exocentric views) and multimodal alignment to improve the performance further, outperforming existing multi-view methods by more than 12\% . MAGLEV emerges as a new state-of-the-art method for fine-grained keystep recognition task, all the while being memory and compute efficient. 2 Related Work Keystep Recognition Ego-Exo4D egoexo4d is the first dataset that provides a unique learning setup to consider large-scale paired first-person (egocentric) and third-person (exocentric) views. It also has various benchmarks, including the task of fine-grained keystep recognition, which we focus on in this paper. An initial study of this task encompasses a diverse set of baseline approaches, such as models learned for action classification (TimesFormer Timesformer_ICML_21 ), video-language pre-training (EgoVLP2 egovlpv2 ), view-invariant two-stage training, view-point distillation and Ego-Exo transfer EgoExo_ICCV21 with an improved backbone. Using both egocentric and exocentric videos during training provides similar or even worse performance when compared to using Ego-view only in the first two baselines, indicating that a more advanced method is required to fully leverage these multi-view data. In this work, we propose a novel framework, MAGLEV , equipped with long-form reasoning, that substantially outperforms these baselines using only egocentric view, and its performance is further improved by utilizing multi-view videos. Our method is based on the recent advancement of graph-based representations for video understanding as described below. Graph-based Representations for Video Understanding A line of work zhao2023constructing ; arnab2021unified ; rai2021home explored scene graphs for video understanding, emphasizing the effectiveness of the structured representations in understanding temporal actions and interactions within videos. It has also been shown that graph-based representations without ground-truth graph annotations can be effective for lightweight frameworks for video understanding applications, including active speaker detection spell2022 ; minintel . In comparison, graph-based representation learning for egocentric videos is a relatively nascent field. Recently, the Ego4D grauman2022ego4d dataset has motivated a few works focusing on egocentric videos. In min2022intel ; min2023sthg , the authors show how the graph-based representation can be leveraged for audio-video diarization in egocentric videos. The task of scene graph generation is addressed in lu2021multi by composing information from third-person and first-person views to construct scene graphs. The work of rodin2024action introduces a temporally evolving graph structure of the actions performed in egocentric videos and proposes a new graph generation task. Our approach is distinguished from the literature in the sense that we formulate the problem of fine-grained keystep recognition as node classification on a graph constructed from the input egocentric videos, while leveraging the variable number