ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning
Jongseo Lee, Kyungho Bae, Kyle Min, Gyeong-Moon Park, Jinwoo Choi · 2025
Read ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning by Jongseo Lee, Kyungho Bae, Kyle Min, Gyeong-Moon Park, Jinwoo Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10896.txt.
Introduction
Jongseo Lee, Kyungho Bae, Kyle Min, Gyeong-Moon Park, and Jinwoo Choi present a 2025 method for teaching video models new classes over time without forgetting the old. Keeping dense clips of past classes eats memory, while sparse clips lose the timing that makes actions readable. Their system splits memory in two, like the mind does.
One store holds sparse features of past clips, the facts of single moments. The other holds broad prompts, the gist of each class. A retrieval module blends the two with cross-attention, rebuilding dense features from sparse ones on demand. Across six standard test sets, the method matched or beat rivals while using far less memory. The authors name the split of memory as the key move.
Catalog record
| CREATOR | Jongseo Lee, Kyungho Bae, Kyle Min, Gyeong-Moon Park, Jinwoo Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10896
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning Jongseo Lee, Kyungho Bae, Kyle Min, Gyeong-Moon Park, Jinwoo Choi Abstract In this work, we tackle the problem of video class-incremental learning (VCIL). Many existing VCIL methods mitigate catastrophic forgetting by rehearsal training with a few temporally dense samples stored in episodic memory, which is memory-inefficient. Alternatively, some methods store temporally sparse samples, sacrificing essential temporal information and thereby resulting in inferior performance. To address this trade-off between memory-efficiency and performance, we propose EpiSodic and SEmaNTIc memory integrAtion for video class-incremental Learning ( ESSENTIAL ). ESSENTIAL consists of episodic memory for storing temporally sparse features and semantic memory for storing general knowledge represented by learnable prompts. We introduce a novel memory retrieval (MR) module that integrates episodic memory and semantic prompts through cross-attention, enabling the retrieval of temporally dense features from temporally sparse features. We rigorously validate ESSENTIAL on diverse datasets: UCF-101, HMDB51, and Something-Something-V2 from the TCD benchmark and UCF-101, ActivityNet, and Kinetics-400 from the vCLIMB benchmark. Remarkably, with significantly reduced memory, ESSENTIAL achieves favorable performance on the benchmarks. 1 1 footnotetext: Equally contributed first authors. 2 2 footnotetext: Corresponding authors. 1 Introduction With advances in deep learning, action recognition models trained on large annotated datasets have achieved remarkable performance [ 7 , 29 , 15 , 4 , 2 , 60 , 48 , 64 , 34 , 26 ] . However, in more realistic scenarios such as class-incremental learning (CIL), action recognition models face significant challenges. In the CIL setting, data for new action classes of interest is incrementally introduced, while data for previously learned classes is not available. Deep learning models tend to overfit to the currently available data, leading to performance degradation on previously learned classes, a phenomenon known as catastrophic forgetting [ 31 , 18 ] . Figure 1 : Performance-memory plot on the UCF-101 [ 46 ] dataset from the TCD benchmark [ 35 ] . ESSENTIAL consistently outperforms existing methods across various memory sizes, achieving a superior performance-memory-efficiency trade-off. We can mitigate catastrophic forgetting by rehearsal training with a few video samples or feature vectors stored in episodic memory. As shown in Figure 2 (a), many prior methods store temporally dense samples, e.g . 16 frames per sample, in episodic memory to preserve temporal information of previous tasks [ 35 , 53 , 52 , 38 ] . However, storing such temporally dense samples is memory-inefficient. Memory-efficiency is crucial in real-world settings as we should consider the scalability, privacy, and legal concerns, as well as the performance of the methods. A potential solution to improve memory-efficiency is to store temporally sparse samples, e.g . 1 or 2 frames per sample, in episodic memory [ 1 , 37 ] . However, as shown in Figure 2 (b), this approach can lead to performance degradation as temporally sparse samples may lack temporal details essential for recognizing actions. We desire a VCIL method that optimally balances high temporal resolution with memory-efficiency, achieving a better trade-off between memory-efficiency and performance. To achieve a better trade-off between memory-efficiency and performance, it is worth considering how humans achieve efficient memory retrieval. Rather than relying on a single memory system, the human brain integrates multiple complementary forms of memory (e.g., episodic and semantic) that interact to enable robust and efficient recall [ 59 , 32 , 12 ] . This highlights that supporting episodic memory with an additional memory component is crucial for achieving a better trade-off between memory-efficiency and performance in VCIL. Figure 2 : MR module achieves a better performance-memory-efficiency trade-off. (a) Storing temporally dense features in episodic memory achieves high performance but results in low memory-efficiency. (b) Storing temporally sparse features improves memory-efficiency but sacrifices performance due to the lack of temporal context. (c) ESSENTIAL tries to combine the best of both worlds. We only store temporally sparse features and lightweight prompts to maintain high memory-efficiency. The MR module integrates episodic and semantic memory to retrieve temporally dense features. The distance between the retrieved feature vector and the original temporally dense feature vector is significantly smaller than the distance between the temporally sparse feature vector and the temporally dense feature vector. ESSENTIAL uses these high-quality retrieved features for rehearsal training to mitigate catastrophic forgetting with high memory-efficiency. Inspired by this, we propose a novel method, E pi S odic and SE ma NTI c memory integr A tion for video class-incremental L earning ( ESSENTIAL ) designed to achieve a better trade-off between memory-efficiency and performance. To reduce memory consumption, ESSENTIAL consists of two efficient memory components: i) episodic memory, which stores temporally sparse features, and ii) semantic memory * * * Used as a methodological term in this work, not strictly equivalent to semantic memory in cognitive neuroscience. , which stores general and abstract knowledge represented by lightweight learnable vectors, referred to as semantic prompts in our method. Notably, we deliberately share a semantic prompt across multiple samples to promote the learning of general knowledge. To effectively mitigate forgetting and achieve high performance, we introduce a novel memory retrieval (MR) module. During rehearsal stage , the MR module retrieves temporally dense features by integrating temporally sparse features from episodic memory with the lightweight prompt from semantic memory through cross-attention. To enable the retrieval capabilities of the MR module, we employ static and temporal matching loss functions during each incremental training stage . Employing the matching loss functions encourages the retrieved temporally dense features to closely match the original temporally dense features. As illustrated in Figure 2 (c), the distance between the retrieved features and the original temporally dense features is significantly smaller than the distance between the temporally sparse features and the original temporally dense features. By effectively retrieving temporal information from temporally sparse features, the MR module enables a high memory-efficiency-performance trade-off as demonstrated in Figure 1 . To validate the effectiveness of ESSENTIAL , we carefully design a set of controlled experiments. We evaluate each component of the proposed method on the temporal-biased Something-Something-V2 (SSV2) dataset [ 19 ] . We compare the proposed method against existing VCIL methods using diverse standard benchmarks: UCF-101 [ 46 ] , ActivityNet [ 6 ] , and Kinetics400 [ 23 ] from the vCLIMB [ 53 ] benchmark [ 53 ] , as well as UCF-101 [ 46 ] , HMDB51 [ 25 ] , and SSV2 [ 19 ] from the TCD benchmark [ 53 ] . To facilitate future research, we plan to release our code. To summarize, our major contributions are as follows: • We propose ESSENTIAL , a memory-efficient video class-incremental learning method that incorporates episodic memory for storing temporally sparse features and semantic memory for storing general knowledge. • To achieve high performance , we introduce a memory retrieval (MR) module that integrates episodic and semantic memory through cross-attention, enabling the retrieval of temporally dense features by leveraging both temporally sparse features and general knowledge stored in semantic memory. 2 Related Work Action recognition. There has been remarkable progress in video action recognition with deep learning over the past decade. Early approaches primarily use CNNs, including 2D [ 10 , 22 , 29 , 44 , 33 , 67 , 44