Keystep Recognition using Graph Neural Networks

Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh · 2025

Read Keystep Recognition using Graph Neural Networks by Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01102.txt.

Introduction

Julia Lee Romero and co-authors teach computers to spot the key steps inside first-person videos of hands-on tasks. Such recognition helps with training aids and assistive tools, but steps unfold over long spans that strain normal video models. Their framework, GLEVR, turns the task into labeling nodes on a graph, where each clip of the video is one node.

The graphs stay sparse and cheap to run, yet the method beat larger existing models by a wide margin. Training also draws on third-person videos of the same tasks and on automatic captions, added as extra nodes. The team tried several ways to wire the nodes together. On the Ego-Exo4D benchmark, the graph approach clearly outran prior methods for fine-grained step recognition.

Catalog record

CREATORJulia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Keystep Recognition using Graph Neural Networks Julia Lee Romero, Kyle Min, Subarna Tripathi, Morteza Karimzadeh Abstract We pose keystep recognition as a node classification task, and propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos. Our approach, termed GLEVR, consists of constructing a graph where each video clip of the egocentric video corresponds to a node. The constructed graphs are sparse and computationally efficient, outperforming existing larger models substantially. We further leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos, as well as adding automatic captioning as an additional modality. We consider each clip of each exocentric video (if available) or video captions as additional nodes during training. We examine several strategies to define connections across these nodes. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods. 1 Introduction (a) Egocentric Vision Graph (b) Multi-view Vision Graph (c) Egocentric Multimodal Graph Figure 1 : GLEVR : Graph-based representation learning for keystep recognition. Learning leverages several strategies such as utilizing additional exocentric views via multi-view alignment, and incorporating complimentary information from other modalities. The inference is always over the egocentric view only. We develop a compute-efficient approach for fine-grained keystep recognition on procedural egocentric videos that leverages long-term dependencies more efficiently and improves egocentric-only predictive performance. Our framework also allows for further improvement by incorporating a variable number of exocentric videos available only during training, and optionally incorporating automatic captioning as an additional input modality as demonstrated in Figure 1 . To summarize, below is the list of our contributions. • Graph-based representation learning for long-videos: We propose a compute-efficient, graph-based representation learning framework for modeling temporal dynamics in long-form videos. We formulate keystep recognition as a node classification problem. We refer to this framework as GLEVR , Graph Learning on Egocentric Videos for keystep Recognition. • Multi-view alignment with graph: GLEVR fuses complementary information from a variable number of views by learning multi-view alignment. Leveraging a variable number of views (if present) during training as one sample makes GLEVR stand out from the baselines, which treat them as separate samples. • Heterogenous graph learning using multimodal alignment: We also present a study on how to leverage complementary multimodal information using GLEVR -Hetero framework. We observe that GLEVR -hetero with video narrations improves accuracy compared to its visual-only counterpart. • Extensive experiments on Ego-Exo4D: GLEVR notably outperforms existing methods by more than 16\% and 12\% , on the validation and test set, respectively. 2 Related Work Keystep Recognition on the Ego-Exo4D Dataset Prior work on this task evaluates a diverse set of baseline approaches, such as models learned for action classification (TimesFormer [ 2 ] ), video-language pre-training (EgoVLP2 [ 4 ] ), view-invariant two-stage training, view-point distillation and Ego-Exo transfer [ 11 ] with an improved backbone. Using both egocentric and exocentric videos during training provides similar or even worse performance when compared to the already-low metrics on Ego-view only in the first two baselines, indicating that a more advanced method is required to fully leverage these multi-view data. Graph-based Representations for Video Understanding A line of work [ 5 , 1 , 17 ] explored scene graphs for video understanding, emphasizing the effectiveness of the structured representations in understanding temporal actions and interactions within videos. It has also been shown that graph-based representations without ground-truth graph annotations can be effective for lightweight frameworks for video understanding applications, including active speaker detection [ 16 , 15 ] . In comparison, graph-based representation learning for egocentric videos is a relatively nascent field. Recently, the Ego4D [ 8 ] dataset has motivated a few works focusing on egocentric videos. In [ 13 , 14 ] , the authors show how the graph-based representation can be leveraged for audio-video diarization in egocentric videos. The work of [ 18 ] introduces a temporally evolving graph structure of the actions performed in egocentric videos and proposes a new graph generation task. Our approach is distinguished from the literature in the sense that we formulate the challenging problem of fine-grained keystep recognition as node classification on a graph constructed from the input egocentric videos, while leveraging the variable number of exocentric videos only during training. 3 Experiments 3.1 Problem Formulation Ego-Exo4D’s keystep recognition task is classification on trimmed video clips [ 3 ] . At training time, we are given egocentric videos and their aligned exocentric videos, while at test time, only the egocentric videos and the trimmed clip segments are provided. 3.2 Graph Architecture We investigate components of the graph architecture, including edge connections, context-length for long-form reasoning, ability to leverage ego-exo relations, and incorporation of multimodal data, specifically text descriptions of segments. We construct a graph given the egocentric input video and keystep segments within the video. Each node corresponds to a keystep segment, and temporal edges connect subsequent node segments. The model is trained on a node classification task, such that a keystep prediction is made for each node in the graph. In this setup, there is only one node type (vision) and one edge type (temporal). Building on the egocentric vision graph, we define a multiview graph, which adds a node for each exocentric view for each keystep segment, drawing ego-exo edges between nodes of corresponding segments. The graph framework also lets us utilize complementary multimodal information such as video narrations. We fine-tune the recent VideoRecap [ 10 ] framework on Ego-Exo4D to generate text narrations for 4-second clips spanning the video length and summarize these narrations for keystep segments using LlaMA-3-8B fine-tuned for summarization [ 19 ] . We extracted LongCLIP [ 21 ] features from the generated segment-level captions. All prior approaches operate directly on video while GLEVR uses pre-extracted visual features, so we evaluate a simple multi-layer perceptron (MLP) on these visual features as an additional baseline comparison. We use the pre-extracted Omnivore Swin-L vision features on each video [ 6 ] , provided by Ego-Exo4D. 3.3 Experimental Setup Dataset We use the full Ego-Exo4D dataset for the fine-grained keystep recognition task [ 3 ] in this work. This includes 1088 videos taking a total of 87 hours, with an average of 4.8 minutes per video, and 289 classes. In our graph setup, the number of graphs in the ego-only and the multi-view setups are equal since all available views are used in one graph. Each node in the graph is used during training, so the multi-view setup has more samples than the ego-only setup. Training and Evaluation For training and evaluation, we employ cross-validation and split the graphs into five splits. We report the validation performance as the average accuracy on the five splits. The framework is trained on each set of 4 splits and evaluated on the remaining split. In accordance with the Ego-Exo4D keystep recognition task definition, we evaluate top-1 keystep recognition accuracy and also report an F1 score at the threshold of 0.1 ([email protected]). 3.4 Experimental Results Method Narration Val Acc Test Acc TimeSFormer [ 2 ] (K600) ✗ 35.25 35.93 EgoVLPv2 [ 4 ] (EgoExo) ✗ 38.21 38.69 VI Encoder [ 20 ] (EgoExo) ✗ 40.23

Read the complete text