Improving action segmentation via explicit similarity measurement
Kamel Aouaidjia, Wenhao Zhang, Aofan Li, Chongsheng Zhang · 2025
Read Improving action segmentation via explicit similarity measurement by Kamel Aouaidjia, Wenhao Zhang, Aofan Li, Chongsheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10713.txt.
Introduction
Kamel Aouaidjia and three co-authors improve action segmentation in video in a 2025 paper. The task is to label each frame with the action it shows and find where one action ends and the next begins. Most methods lean on frame-wise classification, so weak first guesses blur the boundaries.
Their method, ASESM, measures similarity between frames and predictions directly. Transformer encoders make several frame-wise predictions, which are merged by voting. A boundary correction step then nudges borders using feature similarity, and convolution stages polish the result. An unsupervised version finds boundaries from similarity alone. Tests on three cooking and activity data sets, including 50Salads and Breakfast, showed the approach works well.
Catalog record
| CREATOR | Kamel Aouaidjia, Wenhao Zhang, Aofan Li, Chongsheng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10713
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving action segmentation via explicit similarity measurement Kamel Aouaidjia, Wenhao Zhang, Aofan Li, Chongsheng Zhang Abstract Existing supervised action segmentation methods depend on the quality of frame-wise classification using attention mechanisms or temporal convolutions to capture temporal dependencies. Even boundary detection-based methods primarily depend on the accuracy of an initial frame-wise classification, which can overlook precise identification of segments and boundaries in case of low-quality prediction. To address this problem, this paper proposes ASESM (Action Segmentation via Explicit Similarity Measurement) to enhance the segmentation accuracy by incorporating explicit similarity evaluation across frames and predictions. Our supervised learning architecture uses frame-level multi-resolution features as input to multiple Transformer encoders. The resulting multiple frame-wise predictions are used for similarity voting to obtain high quality initial prediction. We apply a newly proposed boundary correction algorithm that operates based on feature similarity between consecutive frames to adjust the boundary locations iteratively through the learning process. The corrected prediction is then further refined through multiple stages of temporal convolutions. As post-processing, we optionally apply boundary correction again followed by a segment smoothing method that removes outlier classes within segments using similarity measurement between consecutive predictions. Additionally, we propose a fully unsupervised boundary detection-correction algorithm that identifies segment boundaries based solely on feature similarity without any training. Experiments on 50Salads, GTEA, and Breakfast datasets show the effectiveness of both the supervised and unsupervised algorithms. Code and models are made available on Github 1 1 1 https://github.com/adjkamel/segsim . [orcid=0000-0001-6286-9527] keywords ,Supervised action segmentation ,Explicit similarity measurement ,Boundary correction ,Fully unsupervised segmentation † † credit: Conceptualization, Experiments, Writing original draft † † credit: Implementation, Experiments † † credit: Implementation, Experiments † † credit: Writing, Reviewing and Editing † † corresponding: Corresponding author. E-mail: [email protected] (C. Zhang) 1 Introduction Action segmentation in videos is essential for many applications such as surveillance, sports analysis, and medical assistance [ 23 , 64 , 61 , 35 ] . It involves the semantic interpretation of distinct events and their temporal location in an untrimmed video. The challenge lies in distinguishing between different actions that occur within the same background, especially when the differences between consecutive actions are minimal, such as with the actions ‘cutting tomato‘ and ‘placing tomato into a bowl‘, where both actions have the same background and objects such as hands and tomatoes. The only difference is using the bowl instead of the knife in the scene. These subtle differences require an efficient method that captures local detail changes as well as global scene variations through learning discriminative frame-level representations and capturing similar patterns of the same action across frames. The earliest action segmentation techniques relied on modeling frame sequences using Hidden Markov Models (HMM), Gaussian Mixture Model (GMM), or a combination of both [ 25 , 26 , 44 ] . With the emergence of deep learning, Recurrent Neural Networks (RNNs) with its variants are widely used for modeling action sequences, including unidirectional RNN, bi-directional RNN, Gated Recurrent Units (GRUs) and LSTM [ 38 , 21 , 45 ] . However, they are unable to capture long-term dependencies in longer videos. Recent methods tackle the problem by capturing temporal dependencies in the sequence using self-attention mechanism in Transformer [ 65 , 3 , 68 ] or temporal convolution for both feature extraction and prediction refinement through multiple stages [ 29 , 30 , 48 , 15 , 32 ] , and they showed high performance by just focusing on frame-wise prediction, where the segmentation is the result of accurate classification of consecutive frames. Over-segmentation methods focus on localizing and correcting boundaries in the sequence [ 60 , 22 , 9 , 10 ] . Existing methods have the following limitations: i) Refinement-based approaches [ 65 , 15 , 32 , 2 ] that rely on frame-wise prediction, heavily depend on the classification accuracy of individual frames, meaning that errors in frame-wise prediction can lead to inaccurate segmentation, especially at segments boundaries. Over-segmentation methods attempt to alleviate this ambiguity by incorporating boundary detection. ii) Although methods such as [ 22 , 60 ] introduce an additional branch to predict frame-wise boundary probabilities [ 22 ] or the start and the end of segments [ 60 ] , the boundary detection is also based on model predictions, lacking a mechanism to verify the accuracy of boundary location correctness explicitly. ii) Boundary correction methods depend on the initial frame-wise prediction for boundary localization, which require high-quality classification as a fundamental step. To tackle the previous limitations, in this paper, we propose ASESM (Action Segmentation via Explicit Similarity Measurement), a supervised learning architecture to enhance action segmentation by measuring the similarity between frames explicitly during the training and testing process. The challenge of inaccurate initial prediction is addressed by using frame-level multi-resolution features as input to multiple transformer encoders to capture global details in small scale features and local details in large scales. Unlike Singhania et al. [46] that use multi-resolution features of the input then stack them together for further processing, we process each of the scaled features with an independent encoder to diverse the feature extraction and predictions. The resulting predictions undergo frame-wise similarity voting to identify the most likely correct class. To address the lack of explicit boundary localization, we introduce a new iterative boundary correction algorithm that adjusts boundary locations. Unlike existing methods that rely solely on initial frame-wise prediction for boundary correction, our approach involves the input frame-wise features around the boundary area to measure the similarity between consecutive frames in an iterative process. The boundary correction is applied during each training iteration, adapting the learning process to adjust the boundaries. The corrected prediction is further refined through multiple stages using temporal convolutions. At prediction time, as a post-processing phase, we optionally apply boundary correction again, followed by a segment smoothing technique that removes outlier classes within segments by measuring the similarity between consecutive predictions, to mitigate the impact of wrong frame-wise classification. Beyond supervised action segmentation, we propose a fully unsupervised boundary detection-correction algorithm that leverages the same similarity measurement metrics used in our supervised approach. This unsupervised technique identifies segment boundaries directly from frame-wise features without requiring any training or coarse initial boundaries. Our experimental results show that both supervised and unsupervised algorithms achieve superior or on par accuracy compared to existing approaches. Moreover, the unsupervised algorithm outperforms other techniques, including those utilizing representation learning. One of the main advantages of our supervised similarity measurement method is that it can be integrated into various existing backbones to improve their segmentation accuracy. The ablation studies demonstrate the significant impact of the proposed similarity measurement components on overall performance. The contributions of our work can be summarized as follows: Figure 1: Framework of our supervised learning architecture for explicit similarity measurement. We involve three levels of similarities: Multi-resolution prediction similarity voting, boundary correction based on frame-wise feature similarity, and segment smoothing based on frame-wise prediction similarity. (1) Enhancing action segmentation by explicitly measuring the similarity between features and predictions in a supervised learning framework through i) voting across multiple frame-wise predictions generated form features of different resolutions, ii) boundary correction based on feature similarity, and ii) segment smoothing to remove outlier classes within segments. (2) A new boundary