An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

Arafat Rahman, Shashwat Kumar, Laura E. Barnes, Anuj Srivastava · 2026

Read An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories by Arafat Rahman, Shashwat Kumar, Laura E. Barnes, Anuj Srivastava (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09231.txt.

Introduction

Rahman and three colleagues build a generative model for skeleton pose trajectories that ignores nuisance factors and keeps shape dynamics. Standard variational autoencoders waste capacity on camera angle, subject size, viewpoint, and pace, rather than shape geometry and motion. Their Elastic Shape VAE uses a transported square root velocity field code on Kendall's shape manifold, stripping translations, rotations, global scaling, and rate variability.

The encoder turns skeletal sequences into a low dimensional latent space with the Riemannian logarithm map, while the decoder rebuilds sequences with the exponential map. On gait cycles, the model predicts mobility scores and sorts subjects into healthy and post stroke groups. On the NTU RGB+D set, it handles action recognition, beating standard VAEs and several sequence baselines. The work offers a principled framework for generative models of longitudinal pose data, with better representations and downstream results.

Catalog record

CREATORArafat Rahman, Shashwat Kumar, Laura E. Barnes, Anuj Srivastava
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories Arafat Rahman, Shashwat Kumar, Laura E. Barnes, Anuj Srivastava Abstract Deep generative models provide flexible frameworks for modeling complex, structured data such as images, videos, 3D objects, and texts. However, when applied to sequences of human skeletons, standard variational autoencoders (VAEs) often allocate substantial capacity to nuisance factors—such as camera orientation, subject scale, viewpoint, and execution speed—rather than the intrinsic geometry of shapes and their motion. We propose the Elastic Shape - Variational Autoencoder (ES-VAE), a geometry-aware generative model for skeletal trajectories that leverages the transported square-root velocity field (TSRVF) representation on Kendall’s shape manifold. This representation inherently removes rigid translations, rotations, and global scaling of shapes, and temporal rate variability of sequences, isolating the underlying shape dynamics. The ES-VAE encoder maps skeletal sequences to a low-dimensional latent space incorporating the Riemannian logarithm map, while the decoder reconstructs sequences using the corresponding exponential map. We demonstrate the effectiveness of ES-VAE on two datasets. First, we analyze skeletal gait cycles to predict clinical mobility scores and classify subjects into healthy and post-stroke groups. Second, we evaluate action recognition on the NTU RGB+D dataset. Across both settings, ES-VAE consistently outperforms standard VAEs and a range of sequence modeling baselines, including temporal convolutional networks, transformers, and graph convolutional networks. More broadly, ES-VAE provides a principled framework for learning generative models of longitudinal data on pose shape manifolds, offering improved latent representation and downstream performance compared to existing deep learning approaches. 1 Introduction Skeleton trajectories have become ubiquitous in activity recognition ( 31 ) , gait recognition ( 12 ) , automatic motion assessment ( 16 ) , and disease severity prediction ( 1 ) . Despite the advances in data collection, extracting meaningful features from such high-dimensional data is far from straightforward. Figure 1: Overview of the Elastic Shape VAE architecture. Raw skeleton trajectories are embedded in Kendall shape space by removing translation, scale, and rotation. Temporal alignment via the transported SRVF removes rate variability. The Riemannian VAE encoder maps tangent vectors to a low-dimensional latent code z , and the decoder maps z back through the tangent space and onto the shape manifold via the exponential map. One approach to learn interpretable representations from high dimensional data is to use variational autoencoders (VAE) ( 22 ) . These have proven to be useful in applications ranging from computer vision ( 17 ) to computational biology ( 26 ) . Applied to pose data, though, a standard VAE devotes a large part of its capacity to irrelevant dimensions, e.g. , camera orientation, sensor distance, body size and rate at which the activity is performed. These nuisance variables are unavoidable in naturalistic gait or activity recording setups where participants walk or perform activity at their own pace and starting position ( 35 ) . However, they play no role in deriving inferences about activity classification or labeling. In fact, it is common in traditional shape analysis ( 8 ) to remove certain nuisance transformations before applying statistical analyses. The question is how to perform this removal while involving powerful ML and AI techniques that are often geared towards Euclidean variables. In order to develop an AI-based technique for analyzing shape sequences, while focusing directly on shapes and shape dynamics, we represent skeleton trajectories as stochastic processes in Kendall shape space ( 21 ) . The representation automatically quotients out translations, rotations, and scaling of individual shapes. We pair this with the transported square-root velocity field (TSRVF) framework ( 2 ) on shape manifold to remove the temporal rate variability (activity speed) of sequences. The resulting model, Elastic Shape VAE (ES-VAE), is invariant to these confounders by design while focusing on shape dynamics in form of meaningful latent representations. Figure 1 shows the overview of the full architecture. 1.1 Contributions 1. We introduce Elastic Shape VAE, a Riemannian VAE that full embeds skeleton trajectories – represented by transported SRVFs on Kendall shape space – into low-dimensional latent space. This representation results in complete invariance to shape translations, rotations, scaling, and speed profiles of activities. 2. Using the unit sphere \mathbb{S}^{2} as an example, we show that Elastic Shape VAE approximates non-geodesic, one-dimensional submanifolds better than such popular ideas Euclidean PCA, Euclidean VAE, and tangent-space PCA. This demonstrates that modeling directly on the manifold preserves nonlinear geometric structure that is lost under Euclidean approximations. 3. On a clinical dataset of 155 participants (111 healthy, 44 stroke), we demonstrate that the learned latent dimensions: (1) map to interpretable gait features (stride length, limb stiffness, arm variability), (2) correlate highly with clinical scores, and (3) outperform other classical and deep learning approaches for predicting stroke severity. 4. On a representative subset of activity recognition dataset, the learned latent dimensions similarly outperformed other classical and deep learning approaches in classifying different activites. The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the mathematical framework. Section 4 describes the experimental setup. Section 5 presents the results. Section 6 discusses the findings and limitations, and Section 7 concludes. 2 Related Literature Riemannian and Geometric Deep Learning : Deep generative models have been pushed beyond Euclidean space in several ways. 28 introduced the Riemannian VAE, where the decoder maps latent codes through the tangent space at a base point and then onto the manifold via the exponential map. The network therefore only has to learn the residual, not the full manifold geometry. 5 used inverse posterior covariances to define a Riemannian metric in the VAE latent space and then used geometry-aware sampling to improve generation compared to standard Gaussian sampling. On the shape-modeling side, 4 built diffeomorphic autoencoders for neuroimaging atlas construction. 10 ; 9 developed Riemannian diffeomorphic autoencoders with implicit neural representations. 15 benchmarked neural shape models on 3D femur data, where hybrid explicit-implicit models beat classical statistical shape models. 14 combined Riemannian VAEs with geodesic-aware Transformers for EEG decoding. All of these operate on general manifolds. We instead work on Kendall shape space , which is purpose-built for skeleton data: translation, rotation, and scale invariance come from the geometric construction itself, so there is no need for data augmentation or learned invariances. 34 used a Kendall pre-shape latent representation, learning landmark-like codes from images using VAE. In contrast, our inputs are skeletons, whose joints are already landmarks, so we model them directly in Kendall shape space. Shape Analysis of Skeletal Data : Several works have been done on the shape analysis of skeleton trajectories. 2 used the transported SRVF framework on Kendall shape space for rate-invariant action recognition. 19 extended the idea to 3D gait recognition with functional PCA on Kendall shape space. 13 paired Kendall shape representations with a CNN-LSTM classifier for skeleton action recognition and added a differentiable transformation layer to handle both rigid and non-rigid normalizations. These are all discriminative approaches (classification). We instead integrate shape-space invariances into a VAE, yielding an interpretable latent embedding of skeleton shape trajectories that suppresses nuisance variation. While there have been several papers on AI-based encoding of static shapes, this is the first effort to encode full shape trajectories in a manner that is invariant to all spatio-temporal nuisance variables. Additional related works are

Read the complete text