2by2: Weakly-Supervised Learning for Global Action Segmentation
Elena Bueno-Benito, Mariella Dimiccoli · 2024
Read 2by2: Weakly-Supervised Learning for Global Action Segmentation by Elena Bueno-Benito, Mariella Dimiccoli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12829.txt.
Introduction
Elena Bueno-Benito and Mariella Dimiccoli tackle global action segmentation, the task of grouping frames that show the same action across videos of mixed tasks. Past work dealt mostly with videos of one task, where action order is much the same. Their work came out in twenty twenty-four.
The pair use task labels to teach action features with weak guidance. Their learning plan works on video pairs and checks three things at once. Frames in one video must split by action, while frames across videos and across tasks must join when the action matches. A paired network built on sparse transformers judges if two clips share a task. Tests on two hard video sets beat prior methods.
Catalog record
| CREATOR | Elena Bueno-Benito, Mariella Dimiccoli |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
2by2: Weakly-Supervised Learning for Global Action Segmentation Elena Bueno-Benito, Mariella Dimiccoli Abstract This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods. Figure 1 : Our approach compares video pairs through a Siamese network by using binary labels indicating if the videos belong to the same activity or not. We propose a triadic loss function modelling intra-video discrimination, inter-video and inter-activity associations for clustering actions across videos of different activities. 1 Introduction Action segmentation, the task of classifying each frame of an untrimmed video plays a fundamental role in various applications such as video surveillance, sports analysis, and content-based video retrieval [ 21 , 50 ] . Recently, this task has received significant attention from the research community. The most reliable approaches for action segmentation are fully supervised methods, which require expensive data annotations [ 5 , 6 , 19 , 27 , 32 , 48 ] . The need for more scalable and practical solutions has led to an increasing interest in developing weakly-supervised [ 9 , 30 , 31 , 33 , 40 , 46 , 49 ] and unsupervised techniques [ 7 , 11 , 12 , 14 , 23 , 24 , 26 , 28 , 35 , 37 , 42 , 43 , 45 , 47 ] . Weakly-supervised methods learn to partition videos into action segments using only transcript annotations for each video, typically in the form of actions transcripts (ordered lists of actions) or action sets (unique actions derived from narrations, captions or meta-tags) [ 31 , 40 , 46 , 49 ] . This weakly-supervised paradigm contrasts with unsupervised methods, broadly categorized into three types, depending on the matching objective [ 13 ] : video-level, activity-level, and global-level. Video-level segmentation methods aim to segment a single video sequence into distinct actions without considering the relationships between actions in different videos [ 7 , 16 , 28 , 35 , 47 ] . While they can be effective for practical applications requiring to segment isolated videos one by one, they fail to generalize actions across different videos. Instead, activity-level segmentation methods focus on matching actions across videos that depict the same complex activity [ 14 , 23 , 24 , 26 , 42 , 46 ] . These methods generally perform poorly at video-level unless temporal smoothing within segments is explicitly modelled. In addition, as they assume or estimate a transcript for each video or set of videos belonging to the same activity, their generalization ability to other activities is hampered. Only Ding et al. [ 14 ] directly addressed global-level segmentation railing on complex activity labels to help discover the constituent actions; however, they do not explicitly model the alignment of actions across videos of the same activity. In this paper, we propose a strategy to discover actions across various complex activity videos, offering a broader and more generalized understanding of actions. Our approach does not require knowledge of video transcripts, but only binary labels indicating whether each pair of videos belongs to the same activity. Therefore, as a weakly-supervised method, it occupies a unique position in the spectrum of action segmentation methods. Our solution , depicted in Figure 1 , aims to enhance the clustering of actions in videos on a global scale through the implementation of a Siamese network based on transformers. This network is designed to address the task of determining whether two videos depict the same activity. Instead of using a standard cross-entropy loss, we propose a triadic loss function capturing the temporal dynamics within individual videos, between similar videos, and across various activities. Our contributions are as follows: 1. We propose a novel weakly-supervised framework for the task of global action segmentation that relies on binary activity labels to discover action clusters across videos of different activities. 2. We introduce a transformer-based Siamese architecture, that takes input pairs of videos, determines if they belong to the same activity or not and aligns them temporally if predicts that they depict the same activity. 3. We introduce a triadic loss function that models intra-video action discrimination at the video-level, inter-video and inter-activity action associations at activity and global-level respectively, for robust action understanding. 4. We achieve state-of-the-art results on the Breakfast (BF) and Inria Instructional Videos (YTI) benchmark datasets, demonstrating the method’s effectiveness and generalization ability across activities. 2 Related work 2.1 Action Segmentation For a comprehensive and recent survey on temporal action segmentation tasks, readers are referred to [ 13 ] . Supervised Action Segmentation. Supervised approaches have seen significant advancements over recent years [ 5 , 6 , 19 , 27 , 32 , 48 ] . Recently, UVAST [ 6 ] integrates fully and timestamp-supervised learning paradigms via sequence-to-sequence translation. This method refines predictions by aligning frame labels with predicted action sequences. LTContext [ 5 ] iterates between windowed local attention and sparse long-term context attention, effectively balancing computational complexity and segmentation accuracy. Lastly, FACT [ 32 ] performs temporal modelling at both frame-level and action-level, facilitating bidirectional information transfer and iterative feature refinement. However, being fully supervised, all these methods are not scalable and not suited for real applications. Weakly-Supervised Action Segmentation. Weakly-supervised techniques have been developed to reduce the need for large annotated datasets. These approaches typically learn to partition a video into several action segments from training videos only using transcripts or other human-generated information to generate pseudo-labels for training [ 30 , 31 , 33 , 40 , 46 , 49 ] . Transcripts have been shown to outperform action set-based methods, while timestamp-based approaches achieve the best results. This suggests that higher levels of supervision generally lead to better performance. In recent years, DP-DTW [ 9 ] has advanced weakly-supervised segmentation by training class-specific discriminative action prototypes. This method represents videos by concatenating prototypes based on transcripts and improves inter-class distinction through discriminative losses. Some methods leverage machine learning models to infer video segments, such as TASL [ 30 ] . Recently, more efficient alignment-free methods have been proposed. MuCon [ 40 ] learns from the mutual consistency between two forms of segmentation: framewise classification and category/length pairs. POC [ 31 ] introduces a loss function to ensure the output order of any two actions aligns with the transcript. Conversely, ATBA [ 46 ] propose an approach that incorporates alignment by directly localizing action transitions for efficient pseudo-segmentation generation during training, eliminating the need