Hierarchical Vector Quantization for Unsupervised Action Segmentation
Federico Spurio, Emad Bahrami, Gianpiero Francesca, Juergen Gall · 2024
Read Hierarchical Vector Quantization for Unsupervised Action Segmentation by Federico Spurio, Emad Bahrami, Gianpiero Francesca, Juergen Gall (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17640.txt.
Introduction
Spurio and co-authors present a method, published in 2024, for cutting long videos into meaningful action segments without any labels. The task, called unsupervised action segmentation, must find segments that mean the same thing across many videos. Recent methods learn features and cluster them in one step. They stumble when the same action varies a lot in how it is performed. The team's method quantizes motion features in two stages, building a hierarchy of clusters.
The second stage splits each action cluster into subclusters that absorb the variation. That structure matches how long each segment lasts far better than rivals do. Tests on standard video sets show the gains. Labeled video is costly, so methods that learn from raw footage matter. They can mine the vast stock of unlabeled video that already exists.
Catalog record
| CREATOR | Federico Spurio, Emad Bahrami, Gianpiero Francesca, Juergen Gall |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17640
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hierarchical Vector Quantization for Unsupervised Action Segmentation Federico Spurio, Emad Bahrami, Gianpiero Francesca, Juergen Gall Abstract In this work, we address unsupervised temporal action segmentation, which segments a set of long, untrimmed videos into semantically meaningful segments that are consistent across videos. While recent approaches combine representation learning and clustering in a single step for this task, they do not cope with large variations within temporal segments of the same class. To address this limitation, we propose a novel method, termed Hierarchical Vector Quantization (HVQ), that consists of two subsequent vector quantization modules. This results in a hierarchical clustering where the additional subclusters cover the variations within a cluster. We demonstrate that our approach captures the distribution of segment lengths much better than the state of the art. To this end, we introduce a new metric based on the Jensen-Shannon Distance (JSD) for unsupervised temporal action segmentation. We evaluate our approach on three public datasets, namely Breakfast, YouTube Instructional and IKEA ASM. Our approach outperforms the state of the art in terms of F1 score, recall and JSD. 1 University of Bonn, Germany 2 Toyota Motor Europe, Belgium 3 Lamarr Institute for Machine Learning and Artificial Intelligence, Germany {fspurio, bahrami, gall}@iai.uni-bonn.de, [email protected] Code — https://github.com/FedeSpu/HVQ 1 Introduction Analyzing videos at a temporal level is crucial for applications in domains like healthcare, manufacturing, neuroscience, and robotics. However, fully-supervised methods ( Abu Farha and Gall 2019 ; Huang, Sugano, and Sato 2020 ; Ishikawa et al. 2021 ; Yi, Wen, and Jiang 2021 ; Bahrami, Francesca, and Gall 2023 ) require expensive frame-wise labels, and weakly-supervised methods ( Chang et al. 2019 ; Ding and Xu 2018 ; Kuehne, Richard, and Gall 2017 ; Li, Lei, and Todorovic 2019 ; Richard, Kuehne, and Gall 2017 ; Richard et al. 2018 ; Khan et al. 2022 ; Li, Abu Farha, and Gall 2021 ; Rahaman et al. 2022 ; Fayyaz and Gall 2020 ; Li and Todorovic 2020 ; Richard, Kuehne, and Gall 2018 ; Souri et al. 2022 ; Behrmann et al. 2022 ) still rely on prior knowledge of action classes. In order to discover semantically meaningful classes directly from long untrimmed videos without any labeling, unsupervised approaches for temporal action segmentation have received increasing attention ( Kukleva et al. 2019 ; VidalMata et al. 2021 ; Li and Todorovic 2021 ; Swetha et al. 2021 ; Kumar et al. 2022 ; Tran et al. 2024 ; Xu and Gould 2024 ) . Figure 1: Given a set of long untrimmed video sequences (first row), the goal is to segment each video and cluster the segments across all videos. Our approach for hierarchical vector quantization learns two codebooks {\bf Z} (second row) and {\bf Q} (third row), which results in a hierarchical representation of actions. Early approaches for unsupervised action segmentation ( Kukleva et al. 2019 ; VidalMata et al. 2021 ; Li and Todorovic 2021 ) learn first a representation and then cluster and segment the frames based on the learned representation in a multi-step fashion. More recent approaches ( Swetha et al. 2021 ; Kumar et al. 2022 ; Tran et al. 2024 ; Xu and Gould 2024 ) combine representation learning and clustering in a single step, since representation learning affects clustering and vice-versa. TOT ( Kumar et al. 2022 ) , for example, applies temporal optimal transport for generating pseudo-labels, which are then used to supervise the representation learning. The recent approach ASOT ( Xu and Gould 2024 ) extends TOT by relaxing the constraint of knowing the action order for a video and it improves the performance. However, it introduces implicitly a strong prior on the segment length. It fails to identify very short segments, as shown in Fig. 4 , and the distribution over the segment lengths deviates largely from the ground-truth distribution, as shown in Fig. 3 . The bias is caused by generating pseudo-labels from a noisy frame-to-cluster assignment matrix, which requires using strong priors like a structural prior. Since methods for temporal unsupervised action segmentation should not have a strong bias towards segments of certain lengths, we propose a new metric based on the Jensen-Shannon Distance (JSD), which measures such bias and complements existing metrics like the F1 score. To address this bias, we furthermore propose a novel approach that is based on a novel paradigm compared to previous approaches that either generate pseudo-labels or follow a multi-step approach. Our approach reformulates the problem as a vector quantization problem where a codebook is learned that represents the clusters. It learns an embedding of frames, a codebook and the assignment of frames to the codebook vectors, and thus clusters, jointly. In contrast to previous works, it neither requires the generation of pseudo-labels nor performs the embedding learning, clustering and assignment of frames in three separate steps. To deal with large intrinsic variations of semantically meaningful segments, we explicitly take in consideration the natural composition of actions, i.e. , in order to complete an action it may be also necessary to complete intermediate steps. We thus propose a hierarchical vector quantization approach, termed Hierarchical Vector Quantization (HVQ), that provides a fine-to-coarse clustering as illustrated in Fig. 1 . The quantization is performed at two levels. The first quantization deals with subactions, a fine-grained representation of an action, while the second quantization combines the subaction clusters to obtain the action representation. In this way, each coarse cluster can be represented by a varying number of finer clusters to cope with the intrinsic variation within each coarse cluster. HVQ is the first hierarchical approach for unsupervised temporal action segmentation and the first approach that utilizes vector quantization. We evaluate the proposed approach on the Breakfast ( Kuehne, Arslan, and Serre 2014 ) , YouTube Instructional ( Alayrac et al. 2016 ) and IKEA ASM ( Ben-Shabat et al. 2021 ) datasets. On all the datasets, the proposed approach achieves state-of-the-art results in terms of F1 score, recall, and the Jensen-Shannon Distance (JSD), which measures the similarity of the distributions of the segment lengths of the predicted and ground-truth segments. In summary, our contributions are: • We propose a novel end-to-end architecture for unsupervised temporal action segmentation. The hierarchical approach is able to capture the variability of action clusters and outperforms a non-hierarchical variant by a large margin. • We introduce a novel metric based on the Jensen-Shannon Distance for evaluating the bias in the predicted segment lengths. • Our method achieves state-of-the-art results for unsupervised temporal action segmentation in terms of recall and F1-score, and it is less biased. 2 Related Works Fully-Supervised Action Segmentation. Early fully-supervised action segmentation methods rely on a sliding temporal window and non-maximum suppression ( Karaman, Seidenari, and Bimbo 2014 ; Rohrbach et al. 2012 ) or model the problem with hidden Markov Models (HMM) ( Kuehne, Gall, and Serre 2016 ; Tang, Fei-Fei, and Koller 2012 ) . Richard and Gall (2016) model the probability of action sequences with language and length models. Recent methods mainly use temporal convolutional networks (TCNs) with temporal pooling to capture long-range dependencies, though this can result in some loss of fine-grained temporal information. For this reason, Abu Farha and Gall (2019) introduced a multi-stage