StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning
Huaijie Wang, De Cheng, Guozhang Li, Zhipeng Xu, Lingfeng He, Jie Li, Nannan Wang, Xinbo Gao · 2025
Read StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning by Huaijie Wang, De Cheng, Guozhang Li, Zhipeng Xu, Lingfeng He, Jie Li, Nannan Wang, Xinbo Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13997.txt.
Introduction
Huaijie Wang and co-authors published a computing paper in 2025 on teaching video models new action classes over time without keeping old examples. Video adds time structure that makes forgetting worse. Their framework, StPR, splits the problem: one part guards frame-level meaning, the other routes time patterns through specialist sub-networks. No stored clips are needed at any point.
Stable meaning channels are picked by sensitivity and class value, then gently held in place. A mixture of experts chosen by motion style handles the time side, with no task label required. On three action datasets the method beat prior approaches and stayed interpretable. The authors stress privacy gains from dropping stored examples.
Catalog record
| CREATOR | Huaijie Wang, De Cheng, Guozhang Li, Zhipeng Xu, Lingfeng He, Jie Li, Nannan Wang, Xinbo Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning Huaijie Wang, De Cheng, Guozhang Li, Zhipeng Xu, Lingfeng He, Jie Li, Nannan Wang, Xinbo Gao Abstract Video Class-Incremental Learning (VCIL) seeks to develop models that continuously learn new action categories over time without forgetting previously acquired knowledge. Unlike traditional Class-Incremental Learning (CIL), VCIL introduces the added complexity of spatiotemporal structures, making it particularly challenging to mitigate catastrophic forgetting while effectively capturing both frame-shared semantics and temporal dynamics. Existing approaches either rely on exemplar rehearsal, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling. To address these limitations, we propose Spatiotemporal Preservation and Routing (StPR), a unified and exemplar-free VCIL framework that explicitly disentangles and preserves spatiotemporal information. First, we introduce Frame-Shared Semantics Distillation (FSSD), which identifies semantically stable and meaningful channels by jointly considering semantic sensitivity and classification contribution. These important semantic channels are selectively regularized to maintain prior knowledge while allowing for adaptation. Second, we design a Temporal Decomposition-based Mixture-of-Experts (TD-MoE), which dynamically routes task-specific experts based on their temporal dynamics, enabling inference without task ID or stored exemplars. Together, StPR effectively leverages spatial semantics and temporal dynamics, achieving a unified, exemplar-free VCIL framework. Extensive experiments on UCF101, HMDB51, and Kinetics400 show that our method outperforms existing baselines while offering improved interpretability and efficiency in VCIL. Code is available in the supplementary materials. 1 Introduction Class-Incremental Learning (CIL) li2017learning ; belouadah2021comprehensive ; de2021continual ; masana2022class ; zhang2024center develops models that learn from a sequence of tasks without forgetting previous knowledge, recognizing an ever-growing set of classes without past task data or identifiers. A key challenge is catastrophic forgetting CatastrophicInterference1989McCloskey ; ConnectionistModels1990Ratcliff , where new knowledge overwrites old. While well studied for images, extending CIL to videos—Video Class-Incremental Learning (VCIL) park2021class ; villa2022vclimb —remains underexplored. VCIL differs from CIL by requiring continual learning of new categories while modeling frame-shared semantics and temporal dependencies, unlike CIL’s focus on static images. This spatiotemporal complexity is critical for understanding actions, motion, and scene dynamics in real-world applications like surveillance, driver monitoring, robotics, and lifelogging. Further, memory and privacy constraints often prohibit storing past video data, demanding continual learning without rehearsal. Figure 1 : (a) Prior methods rely on exemplar rehearsal or naively stack video and CIL modules. (b) Our StPR framework explicitly decouples and reuses spatiotemporal semantics to mitigate forgetting. The central challenge of VCIL lies in mitigating catastrophic forgetting while effectively leveraging frame-shared semantics and temporal dynamics to incrementally learn new categories. Existing methods can be broadly categorized into two types, as illustrated in Figure 1 (a): 1) Exemplar-based methods rebuffi2017icarl ; hou2019learning ; douillard2020podnet ; park2021class ; pei2022learning ; villa2022vclimb ; alssum2023just ; liang2024hypercorrelation store a portion of previous data (video clips, frames, or features) and apply rehearsal to reduce forgetting. However, storing exemplars incurs memory and privacy costs and typically emphasizes frame-level learning without explicitly modeling temporal dynamics. 2) CIL-based methods li2017learning ; dhar2019learning ; cheng2024stsp adapt algorithms developed for static images, using techniques like regularization or subspace projection. While avoiding exemplar storage, they often overlook spatiotemporal properties by flattening or underutilizing temporal features. In contrast, our method StPR (Figure 1 (b)) explicitly decouples video features into Frame shared semantics and temporal dynamics, and reuses these decomposed components to enhance the model’s ability to adapt continually, thereby reducing forgetting without storing extensive exemplars. Specifically, we propose a unified, exemplar-free VCIL framework that fully exploits the spatiotemporal nature of videos. Our method integrates both spatial semantic consistency and temporal variation to mitigate forgetting and improve adaptation across tasks. Separately, we introduce: 1) Frame-Shared Semantics Distillation (FSSD). To preserve frame-shared semantics and alleviate forgetting, we quantify the semantic importance of each channel using a combination of semantic sensitivity and classification contribution. This ensures that semantically meaningful and stable channels are preserved, achieving a better trade-off between stability and plasticity. 2) Temporal-Decomposition-based Mixture-of-Experts (TD-MoE). To exploit temporal dynamics for continual adaptation, we decouple task-specific temporal cues for each expert. At inference, expert routing depends solely on the temporal dynamics of the input, without requiring task identities or stored exemplars. This enables dynamic assignment of weights to experts according to the temporal dynamics of the input, facilitating incremental learning of new categories. Our framework uniquely bridges the gap between video-specific spatiotemporal representation and class-incremental adaptation. By disentangling and leveraging both spatial semantic channel consistency and temporal dynamics, it offers an effective and explainable solution for continual video understanding. Our main contributions are: 1) We propose a Frame-Shared Semantics Distillation method (FSSD) that preserves frame-shared, semantically aligned spatial channels through semantic importance-aware regularization, optimizing the stability-plasticity trade-off in continual learningl; 2) We design a Temporal Decomposition based Mixture-of-Experts strategy (TD-MoE) that decomposes spatiotemporal features and uses temporal dynamics for expert combination, enabling task-id-free and dynamic adaptation; 3) We present a unified, exemplar-free VCIL framework that achieves state-of-the-art results on UCF101, HMDB51, and Kinetics400, demonstrating the effectiveness of integrating spatial semantics and temporal dynamics in VCIL. 2 Related Work 2.1 Class-Incremental Learning Class-Incremental Learning (CIL) aims to enable models to continually learn new classes without forgetting previously learned ones. Existing approaches typically fall into three categories: (1) regularization-based methods kirkpatrick2017overcoming ; zenke2017continual ; xiang2022coarse ; zhou2023hierarchical , which constrain parameter updates to preserve prior knowledge, often via knowledge distillation li2017learning ; hou2019learning ; (2) replay-based methods bang2021rainbow ; chaudhry2018riemannian ; rebuffi2017icarl , which store or generate past data to reduce forgetting; and (3) structure-based methods serra2018overcoming ; mallya2018packnet ; mallya2018piggyback ; liang2024inflora ; yu2024boosting , which expand model capacity or isolate task-specific components. Recently, CIL combined with pre-trained vision transformers (ViTs) ermis2022memory ; CODAPromptCOntinual2023Smith ; wang2022dualprompt ; wang2022learning has achieved impressive results by leveraging transferable representations and modularity. Some methods fully fine-tune pre-trained backbones boschini2022transfer ; zhang2023slca , but this is computationally expensive. To address efficiency, parameter-efficient fine-tuning (PEFT) methods have been introduced. Prompt pool-based approaches wang2022learning ; CODAPromptCOntinual2023Smith ; wang2024hierarchical ; zhang2023slca maintain task-specific prompts, while adapter-based methods zhou2023revisiting ; tan2024semantically ; gao2024beyond ; liang2024inflora ; ExpandableSubspace2024Zhou adapt ViTs to new classes with minimal updates. While effective, most CIL strategies were originally developed for static image domains and do not generalize well to video-based scenarios, where temporal dynamics play a critical role. 2.2 Video Class-Incremental Learning Action recognition has been widely explored with 2D CNNs using temporal aggregation lin2019tsm ; wang2016temporal and 3D CNNs for joint spatiotemporal modeling carreira2017quo ; tran2015learning . More recent work focuses on improving temporal sensitivity and efficiency feichtenhofer2020x3d ; fan2020rubiksnet . However, these models are trained in static setups and do not address continual adaptation or forgetting. To address these challenges, Video Class-Incremental Learning (VCIL) extends conventional Class-Incremental Learning (CIL) to spatiotemporal data, introducing additional challenges such as managing temporal variations across tasks. Several recent methods, including TCD park2021class , FrameMaker pei2022learning , and HCE liang2024hypercorrelation , address this setting by storing videos or compressed exemplars. However, these strategies raise concerns related to memory efficiency and data privacy. While SMILE alssum2023just effectively extracts image features from individual frames, it does not explicitly capture temporal information, which may limit its ability to leverage the distinctive decision cues present in video data. Exemplar-free methods such as STSP cheng2024stsp mitigate forgetting via orthogonal subspace projections, but