ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
Dayoung Gong, Suha Kwak, Minsu Cho · 2024
Read ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation by Dayoung Gong, Suha Kwak, Minsu Cho (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04353.txt.
Introduction
Dayoung Gong, Suha Kwak, and Minsu Cho unified two video understanding tasks in 2024. Action splitting labels what a person is doing in each frame of a video, while action anticipation predicts what they will do next. The jobs are clearly related, yet they had always been trained as separate systems. The authors' single diffusion model, ActFusion, learns both at once by treating them as two views of one sequence.
The trick is a masking plan used in training: the late part of each video is hidden and replaced with learnable tokens. So the model must both segment the visible frames and imagine the invisible future in one pass. Tests showed the two skills help each other, and ActFusion set state-of-the-art results across the three standard cooking-activity benchmarks used in this field, beating systems built for either task alone.
Catalog record
| CREATOR | Dayoung Gong, Suha Kwak, Minsu Cho |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation Dayoung Gong, Suha Kwak, Minsu Cho Abstract Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation and action anticipation, jointly using a unified diffusion model dubbed ActFusion. The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner; the visible part is for temporal segmentation, and the invisible part is for future anticipation. To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future. Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation. ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning. 1 Introduction In everyday life, when interacting with people, we anticipate their future actions while recognizing their actions observed in the present and the past. Similarly, for effective human-robot interaction, robotic agents must recognize ongoing actions while anticipating future behaviors. Two essential tasks in computer vision for such a temporal understanding of human actions are temporal action segmentation (TAS) [ 38 , 19 , 32 , 61 , 12 , 65 , 3 , 43 , 63 , 8 ] and long-term action anticipation (LTA) [ 2 , 34 , 25 , 45 ] . The task of TAS aims at translating observed video frames into a sequence of action segments, while the goal of LTA is to predict a plausible sequence of actions in the future based on the observed video frames. These tasks are closely related in terms of understanding the relations between actions; recognizing actions in the present and the past may improve anticipating action in the future, and the ability to anticipate the future may also enhance recognizing observable actions when facing visual ambiguities. Figure 1 : Task-specific models vs. ActFusion (ours). (a) Conventional task-specific models for TAS and LTA. (b) Our unified model ActFusion to address both tasks. (c) Performance comparison across tasks. Tasks-specific models such as DiffAct [ 43 ] for TAS and FUTR [ 25 ] for LTA exhibits poor performance on cross-task evaluations. ActFusion outperforms task-specific models on both TAS and LTA, including TempAgg [ 52 ] , which trains separate models for each task. Note that the performance of ActFusion is the evaluation result of a single model through a single training process. The reported performance represents the average of each task in the original paper or evaluated with the official checkpoint (See Sec. G for details). Despite the apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. While a growing body of work has shown remarkable progress on both TAS and LTA, these methods are primarily designed for one task (see Fig. 1 a) and show poor generalization when applied to the other (e.g., see FUTR [ 25 ] and DiffAct [ 43 ] in Fig. 1 c). Some prior methods tackle both tasks simultaneously, but they rely on task-specific architectures and require separate training for each individual task (e.g., see TempAgg [ 52 ] in Fig. 1 c). In this paper, we introduce ActFusion, a unified diffusion model that addresses TAS and LTA through a single architecture and training process, as illustrated in Fig. 1 b. The core idea of our unification lies in training the model to effectively handle two different parts of the sequence: visible parts for action segmentation and invisible parts for action anticipation. Accordingly, we propose a new anticipative masking strategy in which a late part of the video frames is masked as invisible, and learnable tokens replace it. We also introduce a random masking strategy to help the model accurately identify actions when some parts of the video are ambiguous [ 59 ] , drawing inspiration from previous methods [ 15 , 6 , 28 , 59 ] . ActFusion generates action sequences from Gaussian noise through iterative denoising, conditioned on both visual features and mask tokens. Through this process, the model jointly learns to classify action labels of visual features and predict future actions of mask tokens. As shown in Fig. 1 c, ActFusion achieves superior performance in both action segmentation and anticipation, demonstrating stronger cross-task generalization compared to previous methods [ 43 , 25 , 52 ] . Furthermore, we find that the benchmark evaluation of some prior LTA methods [ 2 , 20 , 25 ] has exploited the ground-truth length of input videos in testing time, which makes the evaluation unrealistic and problematic since the duration of future actions is supposed to be unknown in advance. To address this issue, we conduct comprehensive experiments both with and without ground-truth length information, providing insights into more realistic deployment scenarios. Our contribution can be summarized as follows: 1) We introduce ActFusion, a unified diffusion model that jointly addresses TAS and LTA through a single training process. 2) We present anticipative masking for effective task unification, along with random masking to enhance robustness against visual ambiguities. 3) Comprehensive experimental results show that jointly learning both tasks provides bi-directional benefits. ActFusion achieves state-of-the-art performance on TAS and LTA across benchmark datasets - 50 Salads and Breakfast, and GTEA - demonstrating the effectiveness of the proposed method. 2 Related work Temporal action segmentation (TAS). The goal of TAS [ 19 , 65 , 43 , 63 , 12 , 23 , 5 , 8 , 32 , 3 , 16 ] is to classify frame-wise action labels in a video. Earlier approaches employ temporal sliding windows [ 50 , 33 ] for action segment detection, grammar-based methods [ 37 , 36 ] have been introduced to incorporate a temporal hierarchy of actions during segmentation. Temporal convolution networks [ 38 , 19 ] and Transformer-based models [ 65 , 8 ] are introduced based on deep learning methods. Since learning long-term relations of actions from activity videos is challenging, a series of work has been proposed to develop refinement strategies [ 30 , 32 , 23 , 12 , 57 , 63 , 3 , 24 ] that can be applied to the TAS models [ 19 , 65 ] . Recently, DiffAct [ 43 ] is proposed to iteratively denoising action predictions conditioned on the input video features adopting the diffusion process. In a similar spirit, the proposed ActFusion is based on the diffusion process, focusing on unifying TAS and LTA through anticipative masking. We demonstrate the bidirectional benefits between the two tasks, showing that learning segmentation along with anticipation is effective. Long-term action anticipation (LTA). LTA [ 2 , 1 , 34 , 20 , 52 , 25 , 44 , 26 ] has recently emerged as a crucial task for predicting a sequence of future actions in long-term videos. Initial