UniSTFormer: Unified Spatio-Temporal Lightweight Transformer for Efficient Skeleton-Based Action Recognition

Wenhan Wu, Zhishuai Guo, Chen Chen, Aidong Lu · 2025

Read UniSTFormer: Unified Spatio-Temporal Lightweight Transformer for Efficient Skeleton-Based Action Recognition by Wenhan Wu, Zhishuai Guo, Chen Chen, Aidong Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08944.txt.

Introduction

Wenhan Wu and co-authors wrote a paper from 2025 on a lean model that names actions from body skeleton data. A camera or sensor can track the joints of a person as they move. Naming the act from those tracks serves sports, games, and care. Transformer models lead the field but tend to be heavy, with split blocks for space and time that cost much to run.

Their model folds space and time into a single attention block. That cuts repeat sums while the model still tracks how poses change. A pooling part blends fine local motion with the broad sweep of the whole act. On standard test sets, the model held accuracy close to the best heavy rivals. It used well over half fewer parts and cost well over half less to run. The team offers it for settings where power and size are tight.

Catalog record

CREATORWenhan Wu, Zhishuai Guo, Chen Chen, Aidong Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08944

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniSTFormer: Unified Spatio-Temporal Lightweight Transformer for Efficient Skeleton-Based Action Recognition Wenhan Wu, Zhishuai Guo, Chen Chen, Aidong Lu Abstract Skeleton-based action recognition (SAR) has achieved impressive progress with transformer architectures. However, existing methods often rely on complex module compositions and heavy designs, leading to increased parameter counts, high computational costs, and limited scalability. In this paper, we propose a unified spatio-temporal lightweight transformer framework that integrates spatial and temporal modeling within a single attention module, eliminating the need for separate temporal modeling blocks. This approach reduces redundant computations while preserving temporal awareness within the spatial modeling process. Furthermore, we introduce a simplified multi-scale pooling fusion module that combines local and global pooling pathways to enhance the model’s ability to capture fine-grained local movements and overarching global motion patterns. Extensive experiments on benchmark datasets demonstrate that our lightweight model achieves a superior balance between accuracy and efficiency, reducing parameter complexity by over 58% and lowering computational cost by over 60% compared to state-of-the-art transformer-based baselines, while maintaining competitive recognition performance. Our codes will be publicly available here . † † address: 1 Department of Computer Science, University of North Carolina at Charlotte, USA 2 Department of Computer Science, Northern Illinois University, USA 3 Center for Research in Computer Vision, University of Central Florida, USA {wwu25, aidong.lu}@uncc.edu , [email protected] , [email protected] Figure 1: Top : Comparison of model performance, size, and efficiency on the NTU-60 [ 1 ] X-Sub benchmark. UniSTFormer achieves the best trade-off among accuracy, parameter size, and computational cost. Bottom : Comparison with recent SOTA methods. UniSTFormer reduces both parameters and GFLOPs by over 58% while maintaining competitive recognition performance (91.0%). The marker size reflects classification accuracy. For more detailed comparisons, please refer to Table 1 and Table 4 . 1 Introduction Skeleton-based action recognition (SAR) [ 2 , 3 , 4 , 5 , 6 , 7 ] has emerged as a fundamental task in computer vision. Transformer-based architectures [ 5 , 8 , 9 , 6 , 7 ] have played a significant role in driving the remarkable progress of SAR in recent years. These models capture the complex spatial and temporal dependencies inherent in human skeleton sequences, modeling the dynamic evolution of joint movements over time and the intricate structural relationships among body parts. Despite recent progress, current transformer-based SAR models still face several fundamental limitations that hinder both their efficiency and generalization capacity. In particular, many methods adopt a two-branch design that separately models temporal and spatial dependencies, typically by stacking or aggregating temporal and spatial modules redundantly ( Challenge 1 ). For example, ST-TR [ 5 ] employs sequential spatial-temporal attention blocks, while Hyperformer [ 8 ] integrates hypergraph-based transformer layers to recombine multi-scale dependencies. Although these designs effectively capture dynamic and structural patterns, they inevitably introduce duplicated feature transformations across branches, leading to redundant computations, inflated parameter counts, and increased architectural complexity. As model complexity increases, the interaction between temporal and spatial blocks further complicates optimization and slows inference, limiting their practical deployment. Moreover, many existing transformer approaches adopt overly complex mechanisms to integrate global and local features, often relying on heuristic-based or pre-set fusion strategies ( Challenge 2 ). For example, SkeMixFormer [ 6 ] applies a multivariate topology design to separately process different spatial relations before fusing them, while FreqMixFormer [ 7 ] introduces frequency-domain modulation but employs a dense combination of spatial and frequency attention blocks. While these methods enhance local-global interactions, they also increase computational overhead and restrict scalability. This underscores the need for lightweight yet powerful SAR frameworks that efficiently capture multi-scale spatio-temporal dependencies while reducing redundancy and computational costs. In this work, we introduce a unified lightweight spatio-temporal transformer framework that directly addresses these limitations. To address Challenge 1 , we design a unified attention mechanism that applies joint-wise attention to each frame, thereby integrating spatial and temporal modeling in a single block without explicit temporal modules. This allows our network to preserve full temporal correlations and capture dynamic motion patterns through shared attention and channel refinement. Moreover, we propose a multi-scale pooling attention module that combines global average pooling and adaptive local pooling to generate joint-wise attention maps, enabling the network to adaptively capture both global motion context and subtle local joint dynamics without relying on complex, handcrafted fusion designs ( Tackling Challenge 2 ). Through extensive ablation studies and experiments on standard SAR benchmarks, we demonstrate that our approach achieves a superior balance between efficiency and accuracy, outperforming state-of-the-art methods while maintaining a significantly reduced parameter footprint (see Figure 1 ). Our key contributions are: 1) We propose a unified module that integrates temporal and spatial modeling in a single attention mechanism, reducing redundancy and simplifying the architecture. 2) We introduce a multi-scale pooling attention that fuses global and local features, achieving a better trade-off between capturing coarse and fine-grained actions and maintaining a significantly reduced parameter footprint. 3) Extensive experiments demonstrate that our design achieves competitive accuracy while reducing parameters by over 58% and computational cost by over 60% compared to SOTA models. Figure 2: Overview of the proposed UniSTFormer framework. Left : The model consists of 10 stacked unified spatial-temporal attention blocks built upon topology-aware skeleton encoding and joint embedding. Top-right : Each block integrates separable convolution, multi-scale attention, and channel refinement with residual fusion (Section 2.1 ). Bottom-right : The multi-scale pooling module combines global and local joint representations to compute dynamic attention maps (Section 2.2 ). 2 Method 2.1 Unified Spatial-Temporal Attention Given an input skeleton sequence X\in\mathbb{R}^{N\times C\times T\times V} , where N is the batch size, C is the number of input channels (e.g., 3 for (x,y,z) ), T is the number of frames, and V is the number of joints, our goal is to predict the action class label y\in\{1,\ldots,K\} . To achieve efficient yet expressive modeling, we propose a unified spatial attention mechanism that eliminates the need for explicit temporal modeling blocks. Specifically, the input X is first transformed using a separable convolution, which reduces parameter overhead while preserving spatial structure: \mathcal{F}=\text{Conv}_{\text{sep}}(X),\quad\mathcal{F}\in\mathbb{R}^{N\times C^{\prime}\times T\times V}, (1) where C^{\prime} is the transformed feature dimension. We then compute a dynamic attention map \mathcal{A}\in\mathbb{R}^{N\times V\times V} from \mathcal{F} using a multi-scale pooling attention module (see Section 2.2 ), which captures both global context and local motion features. This dynamic map is fused with a learnable static topology prior \mathcal{A}_{\text{init}} to form the final attention matrix \mathcal{M} , which enhances joint-wise spatial dependencies in a topology-aware manner: \mathcal{M}=\alpha\cdot\mathcal{A}+(1-\alpha)\cdot\mathcal{A}_{\text{init}}, (2) where \alpha is a learnable scalar. The prior \mathcal{A}_{\text{init}} is initialized from the binary adjacency matrix of the skeleton graph, where two joints are connected if there exists a direct kinematic link between them. \mathcal{M} is then applied to each time step t of the feature map \mathcal{F} , denoted as \mathcal{F}_{t} : \mathcal{F}^{\prime}_{t}=\mathcal{F}_{t}\mathcal{M}^{\top},\quad\forall t. (3) To further enhance the channel-wise discriminability, we apply a lightweight yet effective channel refinement strategy inspired by [ 6 , 10 ] . Specifically, we first perform global average pooling to aggregate feature statistics along the temporal and joint dimensions, followed by a 1D convolution to model inter-channel dependencies. A sigmoid gating function is then applied to generate channel-wise attention weights, which are used to rescale

Read the complete text