Condensing Action Segmentation Datasets via Generative Network Inversion

Guodong Ding, Rongyu Chen, Angela Yao · 2025

Read Condensing Action Segmentation Datasets via Generative Network Inversion by Guodong Ding, Rongyu Chen, Angela Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14112.txt.

Introduction

Guodong Ding, Rongyu Chen, and Angela Yao present a way to shrink video data sets used to train action segmentation models, in a paper from 2025. These data sets label each frame of long task videos, and they are large to store. Their method packs a data set into compact codes.

The approach uses what a network has already learned, run in reverse, to build the compact codes. A sampling step picks varied and typical action sequences to cut repeats across videos. On a breakfast task data set, storage fell by more than five hundred times while most performance was kept. The compact set also beat prior methods when used for step by step learning of new tasks.

Catalog record

CREATORGuodong Ding, Rongyu Chen, Angela Yao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14112

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Condensing Action Segmentation Datasets via Generative Network Inversion Guodong Ding, Rongyu Chen, Angela Yao Abstract This work presents the first condensation approach for procedural video datasets used in temporal action segmentation. We propose a condensation framework that leverages generative prior learned from the dataset and network inversion to condense data into compact latent codes with significant storage reduced across temporal and channel aspects. Orthogonally, we propose sampling diverse and representative action sequences to minimize video-wise redundancy. Our evaluation on standard benchmarks demonstrates consistent effectiveness in condensing TAS datasets and achieving competitive performances. Specifically, on the Breakfast dataset, our approach reduces storage by over 500 × \times while retaining 83% of the performance compared to training with the full dataset. Furthermore, when applied to a downstream incremental learning task, it yields superior performance compared to the state-of-the-art. 1 Introduction The effectiveness of data-driven deep neural network models hinges on training with large and diverse datasets. For instance, ImageNet [ 7 ] comprises over 14 million natural images for image classification, and Common Crawl [ 1 ] provides over 5 billion website pages to foster natural language processing tasks. However, the storage, processing, and training costs associated with massive datasets pose significant challenges, especially as data volumes continue to grow. This need for efficiency has driven interest in dataset condensation [ 36 ] , a technique aimed at compressing large datasets into smaller, information-rich subsets. Such techniques have potential applications in downstream areas, including incremental learning and federated learning. The core objective of dataset condensation is to learn a small synthetic dataset from an original large-scale dataset. Ideally, models trained on the synthetic data should perform comparably to those trained on the original. Significant efforts since then [ 4 , 6 , 22 , 35 , 43 ] have been dedicated to dataset condensation for image data. Video, however, remains under-explored. The recent work [ 37 ] condenses static and dynamic information from videos for action recognition in a two-stage framework. First, a synthetic “image” aggregates the static visual information, while a dynamic memory block is learned to capture and supplement the motion and dynamics. Figure 1 : Comparison of action segmentation performance with dataset storage across common action segmentation benchmarks at different scales. Our method effectively reduces dataset storage while retaining competitive performance to the original setup. This work aims to develop an effective dataset condensation approach for temporal action segmentation (TAS) [ 10 ] . TAS is a task that divides videos into segments by assigning labels on a per-frame basis to capture the sequence and duration of distinct actions. The videos are long, often spanning several minutes, creating significant challenges in raw data storage and processing. Dataset condensation for TAS presents unique challenges that are not typically encountered for action recognition datasets. First, TAS requires frame-level predictions rather than assigning a single label to an entire video or segment, as in action recognition. Consequently, dataset condensation for TAS must be capable of restoring the actual temporal resolutions of segments. Directly adapting the existing video condensation approach proposed by [ 37 ] for condensing TAS datasets is non-trivial. [ 37 ] uses a fixed-length frame processing, which cannot handle varying length segments while being able to restore the original temporal resolution. Additionally, the network in [ 37 ] is designed to work with RGB image sequences, while TAS typically uses pre-computed frame features [ 12 , 39 , 21 ] . Second, the action ordering in the video sequences is not rigidly fixed, but there is still a degree of structure or dependency that governs how actions unfold. Some actions can occur flexibly in their sequence, while others must follow a specific progression. As a result, redundant action sequences can emerge, where identical patterns in action order are presented multiple times. This sequence redundancy is not present in AR datasets. This work presents the first study on condensing TAS datasets. We propose a framework that condenses the dataset with a generative model into a set of latent codes. The generative model is first trained on the entire dataset to learn the prior. Then, through network inversion, we optimize a set of latent codes as the condensed data, minimizing the error between the generated and original frame features. Specifically, we use the TCA model proposed by [ 11 ] as our generative model. TCA is a conditional VAE reconstructing frame features while accounting for temporal dynamics. It is well-suited for our task because it compresses and restores the temporal resolution of action segments through its coherence variable. TCA also condenses the feature dimension, as the latent code is more compact than the original feature. As a result, the storage requirement for a single action segment can be significantly reduced to the segment’s latent code. This corresponds to the first challenge of TAS dataset condensation. To address the second challenge of sequence redundancy, we propose a diversity-based sampling strategy grounded in the edit distance criterion. This approach iteratively selects the candidate sequence that maximizes the diversity within the chosen set. Empirically, we observed that TAS models achieve comparable performance with only half of the sequences. Our approach greatly reduces the storage while achieving comparable segmentation performance to the original dataset as shown in Fig. 1 . Contributions. Our contributions are summarized as follows: (1) This work is the first to investigate and propose an effective dataset condensation approach for the temporal action segmentation task. (2) We propose a generative condensation framework that first learns the generative prior on the dataset and then leverages the network inversion to condense data into compact latent codes. In addition, we propose a sampling strategy to further reduce the storage requirements based on sequence diversity. (3) Our approach effectively condenses TAS datasets of varying scales while consistently yielding comparable segmentation performances. Our approach outperforms the state-of-the-art by 10.7% on Breakfast under an incremental setup. 2 Related Work Temporal Action Segmentation. Various approaches have been proposed to tackle the temporal action segmentation task [ 10 ] . Fully supervised methods require dense annotations for each video frame [ 12 , 39 ] . In contrast, semi-supervised methods [ 32 , 8 ] only necessitate dense labels for a subset of the videos, leaving the rest unlabeled. There are also weaker forms of supervision, such as action transcripts [ 17 ] , action sets [ 28 , 19 , 14 ] , timestamps [ 20 , 27 ] , and activity labels [ 9 ] . Additionally, some approaches [ 29 , 30 , 18 ] operate in an unsupervised setting without relying on any action labels. Recent emerging directions include learning TAS incrementally [ 11 ] where procedural activities are learned sequentially and in an online [ 45 , 31 ] fashion. However, the storage burden of these video datasets remains a pressing issue as these videos are long and with rich redundancy. Our work is the first to study the dataset condensation of temporal action segmentation datasets from the perspective of efficient data storage. Dataset condensation. Dataset condensation (DC) is first formally introduced in [ 36 ] . The target is to learn a small set of synthetic data from an original large-scale dataset so that models

Read the complete text