Efficient Continual Learning through Frequency Decomposition and Integration

Ruiqi Liu, Boyu Diao, Libo Huang, Hangda Liu, Chuanguang Yang, Zhulin An, Yongjun Xu · 2025

Read Efficient Continual Learning through Frequency Decomposition and Integration by Ruiqi Liu, Boyu Diao, Libo Huang, Hangda Liu, Chuanguang Yang, Zhulin An, Yongjun Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22175.txt.

Introduction

Ruiqi Liu and colleagues borrow a trick from human vision to help neural networks learn new tasks without forgetting old ones. People read a scene through two channels at once: broad shapes arrive as low-frequency information, while fine detail rides high frequencies. Writing in twenty twenty-five, the team splits images the same way and gives each frequency band its own small network. The combined system plugs into rehearsal methods, which replay saved examples of earlier tasks during new training.

Low-frequency parts carried general knowledge across tasks, while high-frequency parts guarded the details that separate similar classes. Because the networks are small, the design slashes backbone size and peak memory use, and it trains several times faster on edge devices, by the authors measures. Accuracy rose above leading continual learning methods in their tests.

Catalog record

CREATORRuiqi Liu, Boyu Diao, Libo Huang, Hangda Liu, Chuanguang Yang, Zhulin An, Yongjun Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Continual Learning through Frequency Decomposition and Integration Ruiqi Liu, Boyu Diao, Libo Huang, Hangda Liu, Chuanguang Yang, Zhulin An, Yongjun Xu Abstract Continual learning (CL) aims to learn new tasks while retaining past knowledge, addressing the challenge of forgetting during task adaptation. Rehearsal-based methods, which replay previous samples, effectively mitigate forgetting. However, research on enhancing the efficiency of these methods, especially in resource-constrained environments, remains limited, hindering their application in real-world systems with dynamic data streams. The human perceptual system processes visual scenes through complementary frequency channels: low-frequency signals capture holistic cues, while high-frequency components convey structural details vital for fine-grained discrimination. Inspired by this, we propose the Frequency Decomposition and Integration Network (FDINet), a novel framework that decomposes and integrates information across frequencies. FDINet designs two lightweight networks to independently process low- and high-frequency components of images. When integrated with rehearsal-based methods, this frequency-aware design effectively enhances cross-task generalization through low-frequency information, preserves class-specific details using high-frequency information, and facilitates efficient training due to its lightweight architecture. Experiments demonstrate that FDINet reduces backbone parameters by 78%, improves accuracy by up to 7.49% over state-of-the-art (SOTA) methods, and decreases peak memory usage by up to 80%. Additionally, on edge devices, FDINet accelerates training by up to 5 × \times . $\dagger$ $\dagger$ footnotetext: Corresponding Author. 1 Introduction Continual learning (CL) allows machine learning models to adapt to new data while retaining prior knowledge in dynamic environments [ 26 ] . Nonetheless, the absence of previous training data can result in catastrophic forgetting [ 34 ] , since fitting new classes can erase patterns from previous ones, resulting in performance decline. While recent CL methods focus on preventing forgetting, it is crucial to prioritize learning efficiency on resource-constrained edge devices [ 37 ] , such as smartphones, smart cameras, and embedded systems like the NVIDIA Jetson Orin NX. This becomes especially critical in applications constrained by privacy requirements, limited network connectivity, and the need for rapid adaptation [ 36 , 31 ] . Figure 1 : (a) Diagram of the model compression method. (b) Diagram of the input compression method. (c) Diagram of the proposed FDINet. By utilizing frequency decomposition and integration, FDINet compresses both the input and the model. Through the synergistic replay mechanism of high- and low-frequency components, it compensates for the potential performance loss caused by compression. Figure 2 : Continual learning results on Split CIFAR-10 (5 tasks) using ER-ACE [ 5 ] . Images were pre-processed with high-pass and low-pass filters. Left: Gradient-based attention maps indicating model-attended input pixels. Low-frequency features in the images tend to focus on global structures, while high-frequency features focus more on edge textures and are susceptible to noise. Right: Classification accuracy. Despite filtering out a significant amount of image information, the accuracy degradation is not substantial. To mitigate catastrophic forgetting , various methods have been employed: regularization-based methods [ 23 , 41 , 30 , 8 , 22 ] limit changes to key parameters, preserving crucial knowledge from previous tasks; architecture-based methods [ 13 , 24 , 33 , 46 , 54 ] assign distinct parameters to each task or expand the network to prevent interference between tasks; and rehearsal-based methods [ 2 , 4 , 7 , 10 , 9 ] replay stored samples to maintain past knowledge. Rehearsal-based methods have proven to be the most effective [ 4 ] . However, training efficiency and memory footprint are critical concerns for deploying these methods on edge devices. Long training times and limited edge memory render existing methods impractical for real-world edge applications [ 38 , 16 ] . Therefore, prioritizing inherent method efficiency over complex optimizations is essential for efficient CL method design. To address this, some studies have explored training efficiency in rehearsal-based methods. SOTA efficient CL methods can be broadly classified into two categories: model compression [ 47 ] and input compression [ 32 ] . As illustrated in Fig. 1 (a), model compression [ 47 ] reduces the number of network parameters through sparse training. However, this method fails to effectively reduce training memory footprint because the main bottleneck is intermediate activation storage, not parameters [ 11 , 6 , 49 ] . Input compression [ 32 ] , as shown in Fig. 1 (b), reduces memory footprint and speeds up training by decreasing the size of input feature maps. However, this method fails to fully exploit the redundancy in network parameters, leading to extra computational and memory overhead. To the best of our knowledge, no existing method effectively compresses both input and model concurrently. This motivates our exploration of a joint compression method to improve CL efficiency. Research on the human perception system indicates that both low- and high-frequency components are critical for visual processing. Low-frequency components capture global structures for robust classification, while high-frequency components encode local details for fine-grained recognition [ 44 , 21 ] . Fig. 2 shows the impact of image frequency components on CL. Analogous to the human perception system, high-frequency components in CL capture class-specific edges but are sensitive to noise. In contrast, low-frequency components capture global structure, sacrificing some class details and potentially causing catastrophic forgetting . Notably, despite substantial information filtering, accuracy loss remains modest in both low- and high-pass filtering. This observation prompts the question: Could decoupling high- and low-frequency feature memorization accelerate training with smaller inputs and lighter models without sacrificing accuracy? This method may leverage low-frequency components for general class features and high-frequency components to reinforce specific details during replay. To this end, we propose the Frequency Decomposition and Integration Network (FDINet), a novel CL framework that simultaneously optimizes the training efficiency and performance of CL methods. As shown in Fig. 1 (c), FDINet accelerates training and improves accuracy through the decomposition and integration of different frequency components of the image. Specifically, we design two filters to decompose the input feature maps into reduced-size high-frequency and low-frequency input features. These smaller input features are then processed by two lightweight networks to extract their respective features, which are subsequently classified. To promote more efficient utilization of different frequency features, we design feature aggregators to integrate the intermediate high- and low-frequency features from the two lightweight networks. Our rehearsal framework enhances holistic representations via low-frequency preservation and mitigates the forgetting of class-specific information through high-frequency retention, while significantly compressing the model’s input and structure. Therefore, when combined with various rehearsal-based methods, FDINet can significantly reduce their memory footprint, accelerate training, and maintain accuracy, greatly promoting CL on edge devices. In summary, our contributions are as follows: • We propose FDINet, a novel CL framework designed to collaboratively optimize the training efficiency and performance of CL methods. FDINet compresses the network input and structure through the decomposition and integration of image frequencies. • We propose a frequency-decoupled rehearsal strategy that preserves class stability through low-frequency prototypes while enhancing discriminative power via high-frequency details, thereby mitigating compression-induced performance degradation. • FDINet outperforms SOTA methods in extensive experiments. It boosts accuracy by up to 7.49%, slashes backbone parameters by 78%, reduces peak memory by 80%, and accelerates training by 5 \times on edge devices. 2 Related Work Effective Continual Learning. Effective continual learning focuses on addressing the problem of catastrophic

Read the complete text