Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized?

Jianyang Xie, Yitian Zhao, Yanda Meng, He Zhao, Anh Nguyen, Yalin Zheng, CAS, China. University of Exeter, UK · 2025

Read Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized? by Jianyang Xie, Yitian Zhao, Yanda Meng, He Zhao, Anh Nguyen, Yalin Zheng, CAS, China. University of Exeter, UK (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10679.txt.

Introduction

Jianyang Xie and co-authors ask whether graph networks for action recognition carry too many parameters, in a 2025 paper. The catalog creator field ends with affiliation text, so the head byline governs. These networks read skeleton motion well, yet many rival designs score nearly alike once inputs match, a hint of bloat.

Tests under the lottery ticket idea backed the guess: sparse subnetworks match the full models. Their generator trains a sparse form straight from a random dense start. Even with ninety five percent of weights cut, accuracy fell by under one point on four datasets. Mixing sparsity levels did better still, beating the dense model while using a third fewer weights.

Catalog record

CREATORJianyang Xie, Yitian Zhao, Yanda Meng, He Zhao, Anh Nguyen, Yalin Zheng, CAS, China. University of Exeter, UK
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized? Jianyang Xie, Yitian Zhao, Yanda Meng, He Zhao, Anh Nguyen, Yalin Zheng, CAS, China. University of Exeter, UK Abstract Spatial-temporal graph convolutional networks (ST-GCNs) showcase impressive performance in skeleton-based human action recognition (HAR). However, despite the development of numerous models, their recognition performance does not differ significantly after aligning the input settings. With this observation, we hypothesize that ST-GCNs are over-parameterized for HAR, a conjecture subsequently confirmed through experiments employing the lottery ticket hypothesis. Additionally, a novel sparse ST-GCNs generator is proposed, which trains a sparse architecture from a randomly initialized dense network while maintaining comparable performance levels to the dense components. Moreover, we generate multi-level sparsity ST-GCNs by integrating sparse structures at various sparsity levels and demonstrate that the assembled model yields a significant enhancement in HAR performance. Thorough experiments on four datasets, including NTU-RGB+D 60(120), Kinetics-400, and FineGYM, demonstrate that the proposed sparse ST-GCNs can achieve comparable performance to their dense components. Even with 95 % \% fewer parameters, the sparse ST-GCNs exhibit a degradation of < 1 % <1\% in top-1 accuracy. Meanwhile, the multi-level sparsity ST-GCNs, which require only 66 % 66\% of the parameters of the dense ST-GCNs, demonstrate an improvement of > 1 % >1\% in top-1 accuracy. The code is available at https://github.com/davelailai/Sparse-ST-GCN . 1 Introduction Figure 1 : Model comparisons across the NTURGB+D benchmarks. ’Dense’ means dense backbone, the deviations of Top-1 Acc across all the four NTURGB+D benchmarks remain below 2 \% , and the results are independent of the model size. ‘ 80\% Sparse’ means 80\% parameters are masked out, and the sparsity model shows a slight or no degradation in Top-1 Acc when compared with the corresponding dense model. ’Assemble’ means multi-level sparsity ST-GCNs by incorporating the sparse structure at different sparsity levels, the results show a significant improvement compared with the corresponding dense model. Human action recognition (HAR) is an essential topic in video understanding and has a wide range of applications in intelligent surveillance systems [ 26 ] , and human-computer interaction [ 27 ] . Recently, spatial-temporal graph convolution networks (ST-GCNs) [ 51 , 43 , 10 , 11 , 33 , 53 , 40 ] have gained significant popularity. Compared with other methods [ 46 , 48 , 16 , 54 , 31 , 28 , 3 , 21 ] that rely on RGB image sequence [ 5 ] or optical flow analysis [ 44 ] , the ST-GCNs use body pose and movement information directly, and can effectively capture the interactions between body joints, making it more robust against variations of camera viewpoint and video appearance. Although there are many ST-GCN variants with their own merits, their recognition performance does not vary much after aligning the model setting [ 20 ] . As depicted in Figure 1 , the deviations of top-1 Acc across all four NTURGB+D benchmarks [ 37 , 32 ] remain below 2 \% when considering four different backbones [ 51 , 43 , 10 , 18 ] , and the result is independent of the model size. On the other hand, the model assembling has demonstrated promising improvements in HAR performance [ 47 , 13 , 1 , 47 , 29 , 42 , 14 , 15 ] . However, these approaches typically yield larger models with more parameters, posing challenges for deployment on devices with constrained hardware resources. Observing these, we hypothesize that: 1) ST-GCNs are potentially over-parameterized for HAR . 2) Sparse ST-GCNs may enhance the practical application of assembling models by reducing overall model size. We then substantiated and extended these hypotheses through practical experimentation, and generated a multi-level sparsity ST-GCNs model. This model integrates identical backbones at multiple sparsity levels and demonstrates improved HAR performance without increasing parameters. Inspired by lottery ticket hypothesis (LTH) [ 23 , 22 , 55 , 36 , 12 ] , which states that an independent trainable sub-network exists in a randomly initialized dense model, whose performance is compared to the fully-trained network, we substantiate these hypotheses through practical experimentation in Section 4.2 . Our findings reveal that sparse ST-GCNs extracted from fully-trained dense networks achieve comparable performance, even after masking 80 \% of the parameters. Additionally, we uncover two notable drawbacks of the sparse ST-GCNs obtained directly from the original LTH [ 23 ] : (1) the performance heavily relies on the initial network weights, and (2) a mask characterized by a high sparse ratio leads to a substantial loss of information, resulting in performance collapse. Thus, to address the above drawbacks and obtain stable sparse ST-GCNs from random initial weights, we proposed a sparse ST-GCNs generator for the sub-networks extraction from dense ST-GCNs. Firstly, to reduce the reliance on the initial states, the masks in our method are randomly selected or predefined and then kept fixed, while the convolution kernel weights are subject to learning during training. This process enables the conversion of parameters within the randomly selected sub-network into a trainable state. Secondly, to alleviate the loss of information and retain the advantage of the dense network structure, an information compression loss based on the group lasso penalty [ 34 ] was proposed. This penalty loss was utilized to facilitate the information transformation from weights that have been masked to the intended sparse structures. Furthermore, based on the proposed sparse ST-GCNs generator, we generated multi-level sparsity ST-GCNs by integrating the same models at various sparsity levels. The assembled models have demonstrated a substantial enhancement in HAR performance without parameter increases. This introduces a novel way to improve the overall performance of the assembling ST-GCNs while addressing the challenges of model size. By leveraging sparsity, this approach offers flexibility in model design and implementation, allowing for deployment in diverse scenarios and hardware constraints without compromising efficiency or accuracy. We summarize our contributions as follows: • We successfully substantiated that ST-GCNs exhibit a significant degree of over-parameterization by experiments based on the LTH. To the best of our knowledge, this is the first time to demonstrate a notable degree of over-parameterization in ST-GCNs. • We proposed a sparse ST-GCNs generator, which can train stable sparse ST-GCNs from their randomly initialed dense network. Thorough experiments have verified its efficiency across various ST-GCN backbones. The obtained sparse ST-GCNs demonstrate comparable performance to their fully trained dense counterparts. • Based on the sparse ST-GCNs generator, we demonstrated that multi-level sparsity ST-GCNs, incorporating backbones at different sparsity levels, enhance HAR performance while reducing parameters, offering a novel approach to optimizing assembling ST-GCNs’ overall performance and addressing model size concerns. 2 Related Works 2.1 ST-GCNs for skeleton-based HAR GCNs have been widely used for skeleton-based HAR [ 51 , 43 , 10 , 11 , 33 , 53 , 40 , 18 , 50 , 49 ] . Yan et al . [ 51 ] introduced a pre-defined skeleton graph according to the human body’s natural link and proposed the ST-GCN to capture the spatial and temporal patterns from the graph structure. Upon this baseline, some spatial adaptive graph generation methods based on no-local mechanisms were proposed to increase the flexibility of the skeleton graph structure [ 40 , 10 , 11 , 53 ,

Read the complete text