Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment

Kaiyuan Yang, Xupeng Chen, Jiangpeng He, [4pt] Department of Computer Science, Luddy School, Indiana University Bloomington, Bloomington, USA · 2026

Read Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment by Kaiyuan Yang, Xupeng Chen, Jiangpeng He, [4pt] Department of Computer Science, Luddy School, Indiana University Bloomington, Bloomington, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22371.txt.

Introduction

Yang and co-authors fuse two data streams to grade cerebral palsy severity from gait videos. One stream alone misses part of the picture. Their framework joins skeleton dynamics with clinical gait features. A Grad-CAM pass on a skeleton model first flags the most telling body keypoints. One stream then models the skeleton motion, and the other encodes gait features from those points. Cross-attention fuses the streams. The authors report four-level severity accuracy of about 71 percent.

Catalog record

CREATORKaiyuan Yang, Xupeng Chen, Jiangpeng He, [4pt] Department of Computer Science, Luddy School, Indiana University Bloomington, Bloomington, USA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21173
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22371

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment Kaiyuan Yang, Xupeng Chen, Jiangpeng He, [4pt] Department of Computer Science, Luddy School, Indiana University Bloomington, Bloomington, USA Abstract Video-based gait analysis has become a promising approach for assessing motor impairment in children with cerebral palsy (CP). However, existing methods usually rely on either pose sequences or handcrafted gait features alone, making it difficult to simultaneously capture spatiotemporal motion patterns and clinically meaningful biomechanical information. To address this gap, we propose a multimodal fusion framework that integrates skeleton dynamics with contribution-guided clinically meaningful gait features. First, Grad-CAM analysis on a pre-trained ST-GCN backbone identified the most discriminative body keypoints, providing an interpretable basis for subsequent gait feature extraction. We then build a dual-stream architecture, with one stream modeling skeleton dynamics using ST-GCN and the other encoding gait geatures derived from the identified keypoints. By fusing the two streams through feature cross-attention improved four-level CP motor severity classification to 70.86%, outperforming the baseline by 5.6 percentage points. Overall, this work suggests that integrating skeleton dynamics with clinically meaningful gait descriptors can improve both prediction performance and biomechanical interpretability for video-based CP severity assessment. Keywords: Cerebral Palsy, Gait Analysis, Spatiotemporal Graph Convolutional Networks, Multimodal Fusion, Clinical Interpretability Introduction Cerebral palsy (CP) is one of the most common childhood motor disability, affecting approximately 2–3 per 1,000 live births worldwide [ 1 ] , with substantial regional variation in prevalence [ 2 ] and significant economic burden on healthcare systems [ 3 ] . Because gait dysfunction is a core manifestation of CP, scalable and objective video-based gait assessment has become an important direction for screening, follow-up, and treatment evaluation. Traditional clinical assessment remains widely used [ 4 ] , but it is often subjective and depends on expert judgment, which may introduce inter-rater variability and limit sensitivity to subtle changes over time. Instrumented gait analysis can provide detailed biomechanical measurements [ 5 ] , but it usually requires specialized laboratories and expensive motion capture sensor systems, which limits its accessibility. Caregiver-reported questionnaires can reflect daily functional impact [ 6 ] , but they are also indirect and may not provide sufficiently objective measurements. Recent advances in computer vision have made video-based gait assessment a promising alternative [ 7 ] , addressing accessibility and cost barriers in motor functional disease management. A typical pipeline extracts 2D keypoints from walking videos via pose estimation (e.g., OpenPose [ 8 ] ), models spatiotemporal dynamics, and maps them to clinically meaningful outcomes. Kidziński et al. [ 7 ] demonstrated that clinically relevant gait features can be inferred from monocular videos. Zhao et al. [ 9 ] applied ST-GCN with metric learning for CP severity prediction, while Deng et al. [ 10 ] demonstrated interpretable video-based tracking for motor-state quantification in Parkinson’s disease, highlighting the broader applicability of markerless video assessment. Despite these advances, existing video-based methods still face an important gap. Skeleton-based models are effective at capturing spatiotemporal motion patterns directly from pose sequences, but they mainly use raw keypoint trajectories so that they lack of interpretability [ 20 ] , and do not explicitly incorporate the biomechanical descriptors that are commonly used in clinical gait analysis. In contrast, clinical gait features (e.g., joint angles, step length, symmetry indices) are interpretable and clinically grounded, Samadi Kohnehshahri et al. [ 21 ] reported that many machine learning approaches for CP gait data provide limited transparency, and Xiang et al. [ 22 ] highlighted joint-angle and symmetry-related descriptors as key interpretable signals. These findings motivate methods that connect models with clinically meaningful variables. However the low-dimensional summaries of clinical gait features may miss rich temporal information. As a result, these two sources of information are often modeled separately, even though they are naturally complementary. In this work, to address the mentioned gap between two different modalities, we propose a two-step multimodal fusion framework for video-based CP severity assessment (Fig. 1 ). In Step 1 , we apply Grad-CAM [ 11 ] to a pre-trained ST-GCN backbone to identify the body keypoints that contribute most strongly to prediction. These keypoints are then used to guide the extraction of clinically informed gait features. In Step 2 , we build a dual-stream architecture consisting of a skeleton stream for modeling spatiotemporal dynamics and a clinical feature stream for encoding gait features derived from the selected keypoints. The two streams are fused through feature cross-attention. In this study, CP severity is formulated as a four-class GMFCS label [ 12 ] for supervised classification, which is a common standard in CP motor function assessment to evaluate the performance of models. The main contributions of this work are threefold. First, we introduce a Grad-CAM-guided feature selection strategy that links neural network interpretability with clinically informed gait feature engineering. Second, we propose a dual-stream fusion framework that integrates skeleton dynamics with clinical gait features, leading to a 5.6 percentage point improvement in classification accuracy over the baseline model. Third, we provide a systematic comparison of feature selection and fusion strategies, showing that clinically guided feature subsets can be more effective than using a large set of features without selection. Figure 1: Overview of the proposed two-step framework. Step 1 : An ST-GCN backbone is pre-trained and analyzed via Grad-CAM to identify discriminative keypoints, guiding gait feature selection. Step 2 : A dual-stream architecture combines skeleton dynamics (ST-GCN backbone \to global average pooling \to \mathbf{f}_{s}\in\mathbb{R}^{256} ) with extracted clinical gait features (MLP projection \to \hat{\mathbf{f}}_{c}\in\mathbb{R}^{256} ) via feature fusion for GMFCS classification. Related Work Video-Based Gait Assessment for CP. Recent advances in markerless pose estimation have made video-based gait assessment increasingly feasible for children with CP. OpenPose [ 8 ] and HRNet [ 13 ] have been widely used for this purpose, providing the pose sequences needed for downstream gait analysis. Prior work has shown that clinically relevant gait parameters can be inferred from monocular video. Kidziński et al. [ 7 ] pioneered single-camera prediction of clinical gait metrics, and Zhao et al. [ 9 ] further used ST-GCN with metric learning for CP severity prediction (with GMFCS as an evaluation label). Le and Pham [ 24 ] proposed a Transformer-based network for estimating gait parameters from single-view RGB videos, while Arias Valdivia et al. [ 25 ] applied deep learning to postural control analysis for CP subtype classification. Graph Convolutional Networks for Skeleton Analysis. ST-GCN [ 14 ] introduced spatial graph convolutions combined with temporal convolutions for skeleton-based action recognition, and later variants improved representation power through adaptive topologies [ 15 ] , channel-wise topology refinement [ 16 ] , and attention mechanisms [ 17 ] . These methods are effective for modeling motion dynamics, but most focus on latent skeleton representations without explicitly encoding biomechanical descriptors. Jun et al. [ 18 ] showed that combining skeleton features with gait parameters benefits pathological gait recognition, and Jing et al. [ 19 ] reported clinically meaningful accuracy for deep learning-assisted gait parameter assessment. Together, these works support a multimodal direction that jointly leverages temporal dynamics and structured clinical features. Interpretability in Clinical Gait Analysis. Clinical adoption of deep learning for gait analysis remains limited by insufficient interpretability [ 20 ] . Samadi

Read the complete text