Towards Open-World Human Action Segmentation Using Graph Convolutional Networks
Hao Xing, Kai Zhe Boey, Gordon Cheng · 2025
Read Towards Open-World Human Action Segmentation Using Graph Convolutional Networks by Hao Xing, Kai Zhe Boey, Gordon Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00756.txt.
Introduction
Hao Xing, Kai Zhe Boey, and Gordon Cheng published a computing paper in 2025 on segmenting human actions in video when new, unseen actions appear. Most systems only know the actions they were trained on. The team defines this open-world problem formally and builds a framework to spot and outline unfamiliar actions without hand labels. A graph network with a new upsampling decoder reads body-object structure over time.
Training mixes in synthetic odd-one-out data, and a new loss pulls known actions together while pushing strangers apart. On two hand-object datasets, open-set scores beat prior systems by wide margins. Ablation tests confirmed each piece's value. The authors target assistive robots and health monitoring.
Catalog record
| CREATOR | Hao Xing, Kai Zhe Boey, Gordon Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Open-World Human Action Segmentation Using Graph Convolutional Networks Hao Xing, Kai Zhe Boey, Gordon Cheng Abstract Human-object interaction segmentation is a fundamental task of daily activity understanding, which plays a crucial role in applications such as assistive robotics, healthcare, and autonomous systems. Most existing learning-based methods excel in closed-world action segmentation, they struggle to generalize to open-world scenarios where novel actions emerge. Collecting exhaustive action categories for training is impractical due to the dynamic diversity of human activities, necessitating models that detect and segment out-of-distribution actions without manual annotation. To address this issue, we formally define the open-world action segmentation problem and propose a structured framework for detecting and segmenting unseen actions. Our framework introduces three key innovations: 1) an Enhanced Pyramid Graph Convolutional Network (EPGCN) with a novel decoder module for robust spatiotemporal feature upsampling. 2) Mixup-based training to synthesize out-of-distribution data, eliminating reliance on manual annotations. 3) A novel Temporal Clustering loss that groups in-distribution actions while distancing out-of-distribution samples. We evaluate our framework on two challenging human-object interaction recognition datasets: Bimanual Actions and 2 Hands and Object (H2O) datasets. Experimental results demonstrate significant improvements over state-of-the-art action segmentation models across multiple open-set evaluation metrics, achieving 16.9 % 16.9\% and 34.6 % 34.6\% relative gains in open-set segmentation (F1@50) and out-of-distribution detection performances (AUROC), respectively. Additionally, we conduct an in-depth ablation study to assess the impact of each proposed component, identifying the optimal framework configuration for open-world action segmentation. I INTRODUCTION Fig. 1 : Open-World Human Action Segmentation: detecting and temporally localizing both known and unknown actions. Human-object interactions (HOIs) play a pivotal role in understanding human activities, providing essential cues for applications such as assistive robotics, healthcare, and autonomous systems. Unlike traditional action recognition, HOI analysis requires identifying human actions and objects while also localizing and understanding their interactions over time, a task known as action segmentation. Besides that, for real-world deployment, especially collaborative systems, they must recognize known interactions while detecting and adapting to novel actions. Recently, Graph Convolutional Networks (GCNs) have presented promising results of action segmentation, particularly through skeleton-based representations that offer robustness to occlusions and computational efficiency [ 1 , 2 ] . The Pyramid Graph Convolutional Network (PGCN) [ 1 ] improves frame-wise action segmentation through multi-scale feature fusion. However, existing models operate in closed-world settings, where training and testing datasets share the same action categories. This assumption does not hold in real-world scenarios, where models frequently encounter unseen actions, leading to poor generalization in open-world settings. Current approaches to open-world recognition face critical limitations in reflecting true generalization. Existing methods like the Nearest Non-Outlier (NNO) algorithm [ 3 ] depend on human-annotated unknown samples during fine-tuning, while ActionCLIP [ 4 ] leverages CLIP to extend recognition to unseen actions, which is a large pretrained visual-language model with inherent semantic knowledge of unknown classes. Although effective, both strategies introduce biases: NNO assumes unrealistic access to labeled unknowns, and ActionCLIP leverages external knowledge from pretrained models. These create an unfair advantage that diverges from real-world open-world constraints. Recent advances, such as the Uncertainty-Quantified Temporal Fusion Graph Convolutional Network (UQ-TFGCN) [ 5 ] , address Out-of-Distribution (OOD) detection by preserving physical distance in the feature space but overlook inter-class discriminability among OOD actions. To address these issues, we redefine open-world action segmentation as a generalization task where models must recognize and segment unseen actions using only knowledge from training on closed-world classes, eliminating dependencies on external annotations or pretrained language models, as shown in Fig 1 . We discard the traditional softmax-based classification output for unknown actions classification, which inherently biases predictions toward known classes due to its closed-world confidence calibration. Instead, we explore the relations between feature space and Out-of-Distribution classification, propose a novel Temporal Efficient Upsampling decoder that enables explicit modeling of fine-grained relationships between different actions, and leverages K-means clustering on feature embeddings to categorize OOD actions. In doing so, we propose a Temporal Clustering Loss to enforce tighter feature grouping along the temporal dimension, improving the model’s ability to distinguish In-Distribution from OOD samples. Furthermore, we incorporate Mixup-based augmentation to expose the model to diverse scenarios during training. These innovations ensure more effective OOD detection and classification, enabling robust action segmentation in open-world scenarios. Overall, the technical contributions of the paper are: • We formally define the problem of open-world action segmentation and establish a structured workflow. • We propose an Enhanced Pyramid Graph Convolutional Network framework with three key components: (i) a novel Temporal Efficient Upsampling decoder for better fusion of multi-scale spatio-temporal features, (ii) a Temporal Clustering Loss that enhances the temporal feature clustering, and (iii) Mixup-based data augmentation to simulate OOD scenarios and improve generalization. • We conduct extensive experiments on two challenging HOI datasets—Bimanual Actions (Bimacs) [ 6 ] and H2O [ 7 ] and demonstrate that our approach consistently outperforms existing baseline. Additionally, we perform detailed ablation studies to assess the effectiveness of our framework and evaluate alternative design choices. II RELATED WORK II-A Graph Convolutional Network Recently, Graph Convolutional Networks (GCNs) have emerged as a powerful deep learning framework for learning from graph-structured data. Graphs represent non-Euclidean data structures, and conventional neural networks such as Multilayer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), or Recurrent Neural Networks (RNNs) are not inherently designed to effectively learn from graph data, as they are primarily tailored for Euclidean data (e.g., images, text, RGB-D videos). There are two main types of GCNs commonly used: 1) Spectral GCNs, which operate in the spectral domain. Bruna et al. [ 8 ] introduced spectral networks, generalizing CNNs to graphs by leveraging the graph Laplacian spectrum. 2) Spatial GCNs, which operate directly on the graph domain (nodes and edges). For instance, Hamilton et al. [ 9 ] introduced fixed aggregation functions to summarize neighborhood features, while Veličković et al. [ 10 ] proposed Graph Attention Networks (GATs), extending GCNs by incorporating attention mechanisms. In the context of skeleton-based action recognition, including our work, the latter approach is predominantly adopted. Yan et al. [ 2 ] introduced Spatial-Temporal Graph Convolutional Networks (ST-GCN) for skeleton-based action recognition by leveraging spatio-temporal graphs. Shi et al. [ 11 ] extended this with Two-Stream Adaptive Graph Convolutional Networks (2s-AGCN), which adaptively learn joint and bone features. More recently, Myung et al. [ 12 ] introduced Deformable Graph Convolutional Networks, which dynamically learn the most informative joint features in both spatial and temporal domains. II-B Action Segmentation Temporal action segmentation (TAS) divides an untrimmed video into segments, each assigned an action label, akin to semantic segmentation but in the temporal domain. It enables automatic action recognition by identifying action onset, progression, and conclusion. Approaches in TAS generally follow three architectural designs: (i) Encoder-decoder architectures, such as the Temporal Convolutional Network (TCN) by Lea et al. [ 13 ] , which leverages temporal convolutions to model long-range dependencies efficiently. (ii) Multistage architectures, exemplified by the Multi-Stage Temporal Convolutional Network (MS-TCN) by Farha et al. [ 14 ] , which refines predictions iteratively through stacked single-stage TCNs. Filtjens et al. [ 15 ] extended this with MS-GCN, integrating spatial-temporal graph convolutions for skeleton-based inputs. (iii) Transformer-based architectures, such as ASFormer by Yi et al. [ 16 ] , which employs