Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay
Hongsong Wang, Ao Sun, Jie Gui, and Liang Wang, Fellow, IEEE · 2025
Read Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay by Hongsong Wang, Ao Sun, Jie Gui, and Liang Wang, Fellow, IEEE (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20049.txt.
Introduction
Hongsong Wang, Ao Sun, Jie Gui, and Liang Wang teach computers to learn new hand gestures without forgetting old ones, in a paper published in 2025. Most gesture systems handle a fixed set, but real uses keep adding signs. Saving old training videos may be barred by privacy or storage, so the system must learn from what it already knows.
Their framework invents stand-in features for old gestures from stored class summaries, then mixes them with real data for new gestures. A replay step keeps the classifier's weights aligned with old class centers, and a trimmed loss limits distortion from the fake features. A final retraining balances old and new. Tests on two public gesture datasets showed strong retention. The method suits sign language and control systems that grow over time.
Catalog record
| CREATOR | Hongsong Wang, Ao Sun, Jie Gui, and Liang Wang, Fellow, IEEE |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20049
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay Hongsong Wang, Ao Sun, Jie Gui, and Liang Wang, Fellow, IEEE Abstract Gesture recognition is an important research area in the field of computer vision. Most gesture recognition efforts focus on close-set scenarios, thereby limiting the capacity to effectively handle unseen or novel gestures. We aim to address class-incremental gesture recognition, which entails the ability to accommodate new and previously unseen gestures over time. Specifically, we introduce a Prototype-Guided Pseudo Feature Replay (PGPFR) framework for data-free class-incremental gesture recognition. This framework comprises four components: Pseudo Feature Generation with Batch Prototypes (PFGBP), Variational Prototype Replay (VPR) for old classes, Truncated Cross-Entropy (TCE) for new classes, and Continual Classifier Re-Training (CCRT). To tackle the issue of catastrophic forgetting, the PFGBP dynamically generates a diversity of pseudo features in an online manner, leveraging class prototypes of old classes along with batch class prototypes of new classes. Furthermore, the VPR enforces consistency between the classifier’s weights and the prototypes of old classes, leveraging class prototypes and covariance matrices to enhance robustness and generalization capabilities. The TCE mitigates the impact of domain differences of the classifier caused by pseudo features. Finally, the CCRT training strategy is designed to prevent overfitting to new classes and ensure the stability of features extracted from old classes. Extensive experiments conducted on two widely used gesture recognition datasets, namely SHREC 2017 3D and EgoGesture 3D, demonstrate that our approach outperforms existing state-of-the-art methods by 11.8% and 12.8% in terms of mean global accuracy, respectively. The code is available on https://github.com/sunao-101/PGPFR-3/ . I Introduction Hand gesture recognition is an essential research problem with broad applications in fields such as human-computer interaction, virtual reality, and assistive technology. With the development of deep learning, data-driven gesture recognition methods have made significant progress [ 1 , 2 , 3 ] . However, most of these methods are limited to close-set scenarios and lack the ability to handle unseen or novel gestures. In addition, they often rely on a large amount of annotated data, which is often difficult to obtain in practical applications. Class-incremental gesture recognition is an important problem within the field of open-set recognition, where the model needs to continually learn and adapt to new gestures. However, catastrophic forgetting poses a significant challenge in this context. One straightforward approach to address this issue is to retain samples from old classes and mix them with samples of new classes during the subsequent incremental training of the model [ 4 , 5 ] . However, in real-world applications, past data may not always be fully accessible due to privacy and legal constraints associated with user data [ 6 ] . Furthermore, the limited memory space of devices poses another challenge, as the number of stored samples increases significantly with the addition of incremental tasks. Fig. 1 : Illustration of the proposed data-free class-incremental gesture recognition. The main characteristics of this approach are generating pseudo features in an online fashion during the batch training of new class samples, leveraging prototypes and covariance matrices from old classes. Data-free class-incremental learning addresses the problems of data privacy concerns and limited memory space. Typical data-free approaches use model inversion to generate pseudo samples [ 7 , 8 ] or pseudo features [ 9 , 10 ] of old classes, which would eliminate the need to retain a large generator and only the previously trained network. Recently, Aich et al. [ 8 ] introduce a data-free approach for class-incremental hand gesture recognition, leveraging model inversion to generate pseudo samples based on eigenvectors of the covariance matrix and the support vectors of the classifier. However, the samples obtained through model inversion for each task still need to be retained. Notably, this approach stores the inverted samples rather than the original samples, making it highly inefficient. Despite the already significant computational cost of model inversion, the requirement for vast storage to retain these inversion samples escalates as the number of continual tasks increases. Our objective is to efficiently tackle class-incremental gesture recognition using a data-free approach, as shown in 1 . We focus on generating effective pseudo features for old classes, rather than pseudo samples, thereby significantly reducing computational costs. To enhance the representation of past sample distributions, we introduce class prototypes and covariance matrices for old classes. During the batch training of new class samples for each task in continual learning, pseudo features, prototypes, and new features are concurrently fed into the classifier, working together to address the issue of catastrophic forgetting. To this end, we introduce a framework called Prototype-Guided Pseudo Feature Replay (PGPFR) tailored for class-incremental gesture recognition. The PGPFR comprises four crucial components: Pseudo Feature Generation with Batch Prototypes (PFGBP), Variational Prototype Replay (VPR) for old classes, Truncated Cross-Entropy (TCE) for new classes and Continual Classifier Re-Training (CCRT). To generate a diverse set of pseudo features, the PFGBP combines class prototypes from old classes, features of new classes, and batch-specific class prototypes of new classes, which are computed within each batch of data, utilizing efficient geometric transformations. To strike a balance between overconfidence and uncertainty, temperature sharpening is used to control the steepness of probability predictions of pseudo features. The VPR is built on the assumption that the classifier ought to precisely predict the class associated with the prototype of a particular old class. A plain prototype replay loss is introduced, followed by the VPR that incorporates covariances to enhance robustness. The TCE loss aims to mitigate the influence of domain differences of the classifier [ 11 ] . Lastly, we exploit the classifier re-training strategy for class-incremental gesture recognition. It should be noted that the spatial complexity of our model is low, since the pseudo features generated by our approach are discarded immediately after being used in each batch, eliminating the need for their retention. In summary, the main contributions are listed as follows: • We introduce an efficient Prototype-Guided Pseudo Feature Replay (PGPFR) framework for class-incremental gesture recognition. This framework is capable of continually recognizing gestures from new classes while preserving data privacy. • We propose a Pseudo Feature Generation with Batch Prototypes (PFGBP) module that utilizes efficient operations and batch prototypes to generate pseudo features and pseudo labels for old classes in an online fashion. • We propose Variational Prototype Replay (VPR) which leverages class prototypes and covariance matrices to dynamically update the classifier weights of old classes. Additionally, we introduce Continual Classifier Re-Training (CCRT) strategy to further the problem mitigate catastrophic forgetting. II Related work II-A Video-Based Hand Gesture Recognition Hand gestures primarily consist of various combinations of fingers. Earlier works employed shape matching to measure the dissimilarity among different hand shapes. Deep learning methods learn representations directly from raw data and perform recognition in an end-to-end manner. O Köpüklü et al. [ 12 ] introduce a novel hierarchical structure for neural networks, facilitating the efficient deployment of offline-trained CNN architectures using sliding window techniques. The attention mechanism can enable the model to focus on key areas related to gesture recognition in videos or images, thereby improving recognition performance. Consequently, Dhingra et al. [ 13 ] propose an end-to-end trainable 3D attention residual neural network Res3ATN, which stacks attention blocks and can adaptively change attention