TA-GNN: Physics Inspired Time-Agnostic Graph Neural Network for Finger Motion Prediction
Tinghui Li, Pamuditha Somarathne, Zhanna Sarsenbayeva, Anusha Withana · 2025
Read TA-GNN: Physics Inspired Time-Agnostic Graph Neural Network for Finger Motion Prediction by Tinghui Li, Pamuditha Somarathne, Zhanna Sarsenbayeva, Anusha Withana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13034.txt.
Introduction
Tinghui Li and co-authors build a network that predicts how finger joints will move next. Finger motion is hard to forecast because each finger bends in many ways at once. Their 2025 paper serves uses in virtual reality, graphics, robotics, and rehab, where knowing the next pose early can smooth control.
Their design joins a kinematic feature extractor, which filters velocity and acceleration, with a physics-based encoder built on linear kinematics. A graph decoder learns how joints connect across the hand. The model is time-agnostic, so it can give continuous forecasts for any horizon asked. Tests in virtual reality beat prior methods. Interaction researchers get better finger tracking without extra sensors.
Catalog record
| CREATOR | Tinghui Li, Pamuditha Somarathne, Zhanna Sarsenbayeva, Anusha Withana |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13034
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TA-GNN: Physics Inspired Time-Agnostic Graph Neural Network for Finger Motion Prediction Tinghui Li, Pamuditha Somarathne, Zhanna Sarsenbayeva, Anusha Withana Abstract . Continuous prediction of finger joint movement using historical joint positions/rotations is vital in a multitude of applications, especially related to virtual reality, computer graphics, robotics, and rehabilitation. However, finger motions are highly articulated with multiple degrees of freedom, making them significantly harder to model and predict. To address this challenge, we propose a physics-inspired time-agnostic graph neural network (TA-GNN) to accurately predict human finger motions. The proposed encoder comprises a kinematic feature extractor to generate filtered velocity and acceleration and a physics-based encoder that follows linear kinematics. The model is designed to be prediction-time-agnostic so that it can seamlessly provide continuous predictions. The graph-based decoder for learning the topological motion between finger joints is designed to address the higher degree articulation of fingers. We show the superiority of our model performance in virtual reality context. This novel approach enhances finger tracking without additional sensors, enabling predictive interactions such as haptic re-targeting and improving predictive rendering quality. CCS: Human-centered computing Virtual reality CCS: Human-centered computing Systems and tools for interaction design CCS: Computing methodologies Neural networks Figure 1 . The architecture of TA-GNN, which consists of a kinematics-based encoder and a graph-based decoder. The kinematic-based encoder (orange dash-box) consists of the kinematic feature extractor (green background) and the physics-based encoder (yellow background), which is a joint-axis-specific model. The graph-based decoder (blue background) generates the future motion as a whole hand. The auxiliary loss calculation is applied in the green box. 1. Introduction Human motion prediction has many applications in different fields, such as human-computer interaction ( Koppula and Saxena, 2015 ; Gui et al., 2018 ) , autonomous driving ( Chen et al., 2020 ) , human tracking ( Alahi et al., 2016 ; Gupta et al., 2014 ; Bhattacharyya et al., 2018 ) , and virtual reality (VR) ( Gamage et al., 2021 ) . This method allows systems to understand human behavior and generate realistic motion. It also gives people the opportunity to interact with the systems in a more intuitive and immersive way. Human motion prediction can encompass both full-body movements ( Li et al., 2020 ) as well as specific body parts, including the upper limbs ( Ren et al., 2019 ) , lower limbs ( Zhu et al., 2023 ) , and fingers ( Vangi et al., 2023 ; Li et al., 2017 ) . Particularly in VR, hand and finger interactions have become a widely used method for interacting with virtual objects ( Tanaka et al., 2017 ) . Even though finger-tracking plays a vital role in this interaction method, finger-tracking technologies still experience significant errors and delays due to occlusions, lagging, incorrect graphics rendering, and discrete motion blur ( Abdlkarim et al., 2022 ) , thus leading to poor user experiences. In fields like limb or whole-body motion tracking, motion prediction models have been able to improve on such tracking issues ( Gamage et al., 2021 ) . However, the hands and fingers have many degrees of freedom for movement and the nature of finger movement is highly articulated, making the modelling of interactions between hands and fingers more challenging ( Chan et al., 2007 ) . Accurate and continuous finger motion prediction models can assist in pre-rendering of graphics, anticipating user behavior, and real-time correction of errors. Specifically, continuous prediction is vital in the above applications as it can provide flexibility in predictions ( Jayaraman et al., 2018 ) . Unlike models that only predict motion at certain time periods, a continuous time model can predict how a finger moves continuously without additional training steps. This is crucial for smooth movement predictions, especially when a user interacts with virtual objects ( Gamage et al., 2021 ) with their fingers. Since the learned weights of such a model need not be changed with prediction time, we call them time-agnostic models. The set of linear kinematic equations is one of the most powerful tools in physics of free body motion, capable of accurate time-agnostic predictions. Drawing inspiration from this, we design our model architecture to capture the kinematics of finger joint movement. However, each finger joint does not follow these equations individually due to the forces exerted by muscles, changes in human intentions, and interconnections between the joints. This results in the need for non-linear prediction models. In order to extract the kinematic features of each joint (i.e. angular velocity and angular acceleration as shown in Figure 1 green boxes) from historical data, we use the kinematic feature extractor module, which is shown in Figure 1 green background. We use auxiliary loss functions to ensure the accurate calculation of these kinematic features. To enforce kinematic constraints and encode temporal features of each joint we follow this with the time-agnostic physics-based encoder module (Figure 1 yellow background). For learning the spatial domain interconnections between joints, we apply a graph convolutional network as the decoder (Figure 1 blue background) to extract the topological movement information across different finger joints. Given that each finger joint maintains an associative relationship with its parent joint, we set up the edge connections of the decoder to learn and predict how the fingers move in a realistic way. Combining these three components, we derive a time-agnostic deep-learning model called TA-GNN, which is able to learn both the temporal and topological information within and across each finger joint. The architecture of TA-GNN is illustrated in Figure 1 . The numerous large-scale datasets ( Ionescu et al., 2013 ; Punnakkal et al., 2021 ; Mahmood et al., 2019 ) available for whole-body motion prediction have been one of the key contributors to the popularity and rapid advancements in the field. However, in comparison finger motion datasets are limited in number, especially for VR applications ( Jarque-Bou et al., 2020 ; Roda-Sales et al., 2022b ; Yang et al., 2023 ) . Some of the available datasets face constraints in accuracy and generalization as they have been collected using data gloves with strain gauges ( Roda-Sales et al., 2022a ) . We conducted a user study to collect a novel finger motion capture dataset called VRHands dataset in VR context. The experiments conducted on the VRHands dataset as well as existing Re:InterHand ( Moon et al., 2023 ) dataset show that TA-GNN provides significant improvements compared to baseline. We also show that TA-GNN outperforms the methods introduced previously by Chen et al. ( Chan et al., 2007 ) and Yang et al. ( Yang et al., 2023 ) . The main contributions of the paper can be summarized as: • We propose TA-GNN – a physics-inspired time-agnostic graph neural network – based on historical kinematic features and topological movement information. TA-GNN consists of a kinematic feature extractor, a physics-based encoder, and a graph-based decoder; • The experiment results show the proposed TA-GNN significantly outperform baseline models and capable of accurately predicting future movement for time-agnostic instances; • We propose a new direction for improving finger tracking without additional sensors in virtual reality, enabling predictive interactions such as haptic re-targeting and improving predictive rendering quality. 2. Related work 2.1. Human motion prediction One of the