CascadeFormer: A Family of Two-stage Cascading Transformers for Skeleton-based Human Action Recognition

Yusen Peng, Alper Yilmaz · 2025

Read CascadeFormer: A Family of Two-stage Cascading Transformers for Skeleton-based Human Action Recognition by Yusen Peng, Alper Yilmaz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00692.txt.

Introduction

Yusen Peng and one co-author, Alper Yilmaz, propose CascadeFormer for skeleton-based human action recognition, identifying actions in video from sequences of joint coordinates. Graph convolutional networks have dominated this field because skeleton data has a natural space-time structure, but transformers with masked pretraining open new options.

CascadeFormer is a family of two-stage cascading transformers. A masked pretraining stage first learns general skeleton representations by hiding and reconstructing parts of the input, then a cascading fine-tuning stage adapts the model for action classification. Peng and Yilmaz evaluate the family on three benchmarks, Penn Action, N-UCLA, and NTU RGB+D 60, achieving competitive performance throughout, and they release code and model checkpoints.

Catalog record

CREATORYusen Peng, Alper Yilmaz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CascadeFormer: A Family of Two-stage Cascading Transformers for Skeleton-based Human Action Recognition Yusen Peng, Alper Yilmaz Abstract Skeleton-based human action recognition leverages sequences of human joint coordinates to identify actions performed in videos. Owing to the intrinsic spatiotemporal structure of skeleton data, Graph Convolutional Networks (GCNs) have been the dominant architecture in this field. However, recent advances in transformer models and masked pretraining frameworks open new avenues for representation learning. In this work, we propose CascadeFormer , a family of two-stage cascading transformers for skeleton-based human action recognition. Our framework consists of a masked pretraining stage to learn generalizable skeleton representations, followed by a cascading fine-tuning stage for action classification. We evaluate CascadeFormer across three benchmark datasets—Penn Action, N-UCLA, and NTU RGB+D 60—achieving competitive performance on all tasks. We open source our code at https://github.com/Yusen-Peng/CascadeFormer and release model checkpoints at https://huggingface.co/YusenPeng/CascadeFormerCheckpoints. Introduction Human action recognition is a fundamental computer vision task that aims to identify the action performed by a target person in a given video, represented as a sequence of frames. Human actions can be interpreted through various modalities, such as raw RGB images ( Zhang, Gupta, and Zisserman 2021 ) , skeletal joint coordinates ( Zhao et al. 2019 ) , or a fusion of both ( Kim, Ahn, and Ko 2023 ) . Compared to RGB-based approaches, skeleton-based action recognition offers distinct advantages ( Zhou et al. 2023 ) : it is computationally efficient due to relying solely on human joint coordinate data, and it is more robust to environmental noise and variations in camera viewpoints. Using these strengths, skeleton-based action recognition has been explored through various methods, including probabilistic models ( Zhao et al. 2019 ) , recurrent neural networks ( Shahroudy et al. 2016 ; Liu et al. 2016 ) , and graph convolutional networks (GCN) ( Yan, Xiong, and Lin 2018 ) . Nevertheless, the rapid advancement of attention mechanisms ( Vaswani et al. 2023 ) in natural language processing has driven the widespread adoption of transformer architectures in computer vision. The success of vision transformers ( Dosovitskiy et al. 2021 ; Rao et al. 2021 ) across tasks such as image classification ( Ridnik et al. 2021 ) , multimodal learning ( Radford et al. 2021 ) , and visual instruction tuning ( Liu et al. 2023 ) underscores the potential of transformers for skeleton-based action recognition. Recent approaches incorporate novel attention designs tailored to spatiotemporal data ( Bertasius, Wang, and Torresani 2021 ; Zhou et al. 2023 ; Wang et al. 2023 ) ; however, these transformer-based models are predominantly trained in an end-to-end manner. However, instead of relying on end-to-end training—which may risk overfitting on downstream tasks—both language modeling ( Devlin et al. 2019 ) and multimodal learning ( Srivastava and Sharma 2023 ; Srivastava and Sharma 2024 ) have widely embraced masked pretraining as a precursor to supervised adaptation. Motivated by the goal of equipping simple transformers with masked pretraining , we propose CascadeFormer , a family of two-stage cascading transformers for skeleton-based human action recognition. CascadeFormer comprises a lightweight transformer for masked pretraining and an additional transformer for fine-tuning on action labels. In this paper, we begin with a review of prior work in masked pretraining with transformers and skeleton-based action recognition, followed by a brief overview of the input skeleton data format. We present our contributions: 1. We introduce CascadeFormer , a framework comprising three model variants that share a unified pipeline involving masked pretraining followed by cascading fine-tuning. 2. We conducted extensive evaluations of CascadeFormer on three widely used and diverse datasets: Penn Action, N-UCLA, and NTU RGB+D 60. 3. We performed extensive ablation studies—summarized in the main paper and further expanded in the appendix—to analyze the impact of architectural choices and training configurations. Finally, we conclude with a discussion of the implications of our work. Figure 1: Overview of the masked pretraining component in CascadeFormer. A fixed percentage of joints are randomly masked across all frames in each video. The partially masked skeleton sequence is passed through a feature extraction module to produce frame-level embeddings, which are then input into a temporal transformer (T1). A lightweight linear decoder is applied to reconstruct the masked joints, and the model is optimized using mean squared error over the masked positions. This stage enables the model to learn generalizable spatiotemporal representations prior to supervised finetuning. Related Work In this section, we first provide a brief overview of the development of masked pretraining frameworks in various fields. We then review widely adopted approaches in the recognition of human action based on skeletons. Masked Pretraining Framework The masked pre-training was first introduced in the domain of language modeling, most notably with BERT ( Devlin et al. 2019 ) . By randomly masking 15\% of the input tokens and training the model to predict them in a self-supervised manner, BERT can learn deep bidirectional representations ( Devlin et al. 2019 ) . A similar paradigm has been extended to computer vision with masked autoencoders (MAEs) ( He et al. 2021 ) , which apply a significantly higher masking ratio (up to 80\% ), enabling effective self-supervised pretraining for vanilla vision transformers ( Dosovitskiy et al. 2021 ) . More recently, the masked pretraining framework has been rapidly adopted in multimodal learning. Models such as OmniVec ( Srivastava and Sharma 2023 ) and OmniVec2 ( Srivastava and Sharma 2024 ) take advantage of masked auto-encoding techniques to learn unified representations in diverse modalities, including image, text, video, and audio. Skeleton-based Action Recognition Human action recognition can be explored through various modalities, including raw RGB images ( Zhang, Gupta, and Zisserman 2021 ) , skeletal joint coordinates ( Zhao et al. 2019 ) , or a fusion of both ( Kim, Ahn, and Ko 2023 ) . In contrast to RGB-based methods, skeleton -based action recognition is characterized by its computational efficiency — owing to its reliance solely on human joint coordinate data — and its robustness to environmental noise and camera viewpoint variations, since it preserves only the spatiotemporal information of keypoints ( Zhou et al. 2023 ) . Although probabilistic models ( Zhao et al. 2019 ) demonstrated promising performance, deep learning approaches have become dominant in this domain. Given the temporal nature of video sequences, recurrent neural networks (RNN) ( Shahroudy et al. 2016 ; Liu et al. 2016 ) have proven effective in modeling temporal dynamics. Furthermore, convolutional neural networks (CNNs) ( Cai et al. 2023 ; Duan et al. 2022 ) have been adapted to this task by transforming skeleton data into pseudo-images, allowing convolutional operations to be applied. Furthermore, graph-based neural networks—including graph neural networks (GNNs) ( Shi et al. 2019 ) and graph convolutional networks (GCNs) ( Yan, Xiong, and Lin 2018 ) —have gained popularity due to the natural graph structure of human skeletons, where joints and bones are represented as vertices and edges, respectively. More recently, transformer architectures ( Vaswani et al. 2023 ) have attracted increasing attention in skeleton-based action recognition. These models are often enhanced with efficient spatiotemporal attention mechanisms ( Zhou et al. 2023 ; Do and Kim 2024 ; Bertasius, Wang, and Torresani 2021 ; Wang et al. 2023 ) ,

Read the complete text