Improvement of Human-Object Interaction Action Recognition Using Scene Information and Multi-Task Learning Approach
Hesham M. Shehata, Mohammad Abdolrahmani · 2025
Read Improvement of Human-Object Interaction Action Recognition Using Scene Information and Multi-Task Learning Approach by Hesham M. Shehata, Mohammad Abdolrahmani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-09067.txt.
Introduction
Recognizing that someone is using an ATM, not just standing near it, defeats skeleton-only action recognition. Hesham M. Shehata and Mohammad Abdolrahmani add scene awareness in a 2025 research paper. Graph networks reading body poses perform well on general actions. They fail on human-object interactions because the object and its place in the scene are missing from the input.
The authors feed fixed object information from the environment into the model. A multi-task learning setup trains recognition jointly with interaction-area information. They collected real data in public spaces, covering interactions with fixed machines and plain walking or standing. Accuracy rose well above the skeleton-only baseline, reaching near-perfect scores on their dataset. Surveillance, assistive monitoring, and smart buildings all need machines that understand not just how people move, but what they are engaging with.
Catalog record
| CREATOR | Hesham M. Shehata, Mohammad Abdolrahmani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-09067
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improvement of Human-Object Interaction Action Recognition Using Scene Information and Multi-Task Learning Approach Hesham M. Shehata, Mohammad Abdolrahmani Abstract Recent graph convolutional neural networks (GCNs) have shown high performance in the field of human action recognition by using human skeleton poses. However, it fails to detect human-object interaction cases successfully due to the lack of effective representation of the scene information and appropriate learning architectures. In this context, we propose a methodology to utilize human action recognition performance by considering fixed object information in the environment and following a multi-task learning approach. In order to evaluate the proposed method, we collected real data from public environments and prepared our data set, which includes interaction classes of hands-on fixed objects (e.g., ATM ticketing machines, check-in/out machines, etc.) and non-interaction classes of walking and standing. The multi-task learning approach, along with interaction area information, succeeds in recognizing the studied interaction and non-interaction actions with an accuracy of 99.25%, outperforming the accuracy of the base model using only human skeleton poses by 2.75%. Keywords: Human Action Recognition (HAR), Human-Object Interaction (HOI), Scene Information (SI), Multi-Task Learning (MTL), Graph Convolutional Neural Networks (GCNs), Scene Perception Graph Neural Networks (SPGCNs), Gated Recurrent Units (GRUs) 1 Introduction In recent years, there has been a promising development of autonomous security systems with the aid of security cameras fitted in public areas (e.g., shopping malls, parking areas, etc.), which are becoming increasingly popular to ensure high quality of public security. Also, the contribution of machine learning models in the field of action recognition has increased rapidly. However, these systems lack effective representation of the scene information and appropriate learning architectures, so they fail to recognize interaction scenarios effectively (Human-Human Interaction (HHI) or Human-Object Interaction (HOI)). Therefore, to develop our HOI detection model, we collected our video data from public places (e.g., shopping malls, indoor buildings, etc.), performed skeleton pose estimation using Asilla pose, and investigated the use of skeleton pose data only versus combined data with fixed object nodes for learning approaches to recognize human object interaction scenarios (e.g., ATM ticketing machines, check-in/out machines, etc.) effectively while paying attention to the model performance with non-interaction scenarios as well (e.g., walking, standing, etc.). The purpose of this research is to maximize the recognition of human-object interaction scenarios by considering scene information along with following a multi-task learning approach. In this study, we propose a novel approach to improve human-object interaction (HOI) recognition by integrating scene information with a multi-task learning framework. This methodology aims not only to capture the complexity of human-object interactions but also to leverage additional scene-related cues that can enhance the recognition performance. Beyond recognizing interactions, the proposed framework has demonstrated significant improvements in identifying non-interaction scenarios, addressing challenges in real-world settings such as distinguishing between standing still and interacting with an object. To validate the effectiveness of this approach, we evaluated three different architectures of Graph Convolutional Networks (GCNs), each tailored to exploit spatial and temporal dependencies in human poses and scene context. The comparative analysis reveals how incorporating scene information and multi-task learning yields notable improvements over baseline models. The remaining sections are organized as follows: Section 2 discusses the related work. Section 3 proposes the methodology to successfully learn human-object interaction scenarios based on skeleton poses and fixed object nodes. Section 4 presents a case study of human-object interaction/non-interaction scenarios. Section 5 shows the results following the proposed methodology. Section 6 discusses the findings of the case study based on the results. Finally, Section 7 shows our conclusions. 2 Related Work Human-object interaction (HOI) recognition has witnessed substantial progress in recent years, primarily through advancements in graph-based learning architectures, transformer-based methods, and multi-task learning frameworks. These methods aim to address the complexities of modeling spatial, temporal, and contextual relationships in human-object interactions. However, existing approaches still face challenges in integrating scene information and generalizing across diverse environments. 2.1 Transformer-Based Approaches Transformers have recently gained popularity for their ability to model relationships in sequential and spatial data. Key works include: • DETR [ 1 ] : This end-to-end transformer-based object detection model has been adapted for HOI recognition by leveraging attention mechanisms to capture human-object relationships. While highly accurate for RGB-based methods, DETR is computationally intensive and unsuitable for lightweight, skeleton-based models. • Action Genome [ 2 ] : This approach models visual relationships using a scene graph-based framework. While it focuses on HOI recognition, it lacks the ability to handle skeleton-based representations or real-time applications. • VidTr [ 3 ] : A transformer-based model for video understanding that achieves competitive performance on HOI datasets like AVA and HICO-DET. However, its reliance on high-resolution video data makes it impractical for scenarios requiring skeleton-based or resource-efficient models. 2.2 Graph-Based Learning Approaches Graph Convolutional Networks (GCNs) have emerged as a powerful tool for modeling skeletal and contextual data. Recent advances include: • Dynamic GCN [ 4 ] : An adaptive GCN that dynamically learns graph connections based on input data. While effective for action recognition, it struggles to incorporate scene context and is computationally expensive for real-time applications. • 2G-GCN [ 26 ] : This geometric feature-informed GCN achieves 92.4% accuracy on custom HOI datasets by modeling relationships between human poses and objects. However, it does not incorporate interaction-specific scene nodes or temporal dependencies. • SPGCN [ 27 ] : A scene-perception GCN that embeds environmental context into the graph structure. Despite its focus on HOI scenarios, it achieves limited accuracy (78.3%) due to its static graph structure and lack of multi-task learning. 2.3 Multi-Task Learning and Hybrid Architectures Multi-task learning has been explored to improve the generalizability of HOI models: • HAMLET [ 7 ] : A multi-modal, hierarchical, attention-based approach that processes both spatial and temporal features. However, it focuses on unimodal data streams and does not explore interaction-specific scenarios. • ST-GCN [ 8 ] : A spatio-temporal GCN that effectively models temporal dependencies but does not consider multi-task learning or scene information. 2.4 Skeleton-Based HOI Recognition While RGB-based approaches dominate the HOI literature, skeleton-based methods offer lightweight and privacy-preserving alternatives: • Efficient-GCN [ 25 ] : A streamlined GCN architecture optimized for skeleton-based action recognition. However, it overlooks scene interaction nodes, limiting its application to HOI scenarios. • PoseC3D [ 10 ] : A 3D convolution-based model that processes skeletal data for action recognition. Despite its effectiveness, it lacks contextual modeling of scene information. 2.5 Key Gaps in SOTA and Contributions The following limitations are observed in existing methods: • Scene Information: Most SOTA methods fail to integrate scene-specific interaction areas (e.g., fixed objects) into their learning frameworks. • Temporal Dependencies: Limited exploration of hybrid architectures (e.g., GCN+GRU) to model temporal dynamics in HOI scenarios. • Multi-Task Learning: Few approaches simultaneously classify interaction and action types, which limits their generalizability. • Lightweight Models: Many SOTA methods (e.g., DETR, VidTr) are computationally expensive and unsuitable for real-time applications. The proposed method addresses these gaps by: • Embedding scene interaction nodes into the graph structure, enabling a more comprehensive representation of contextual information. • Introducing a hybrid GCN+GRU architecture to effectively capture temporal dependencies in human-object interactions. • Implementing a multi-task learning framework for simultaneous classification of interaction and action types, reducing overfitting and improving