CD 2 : Constrained Dataset Distillation for Few-Shot Class-Incremental Learning
Kexin Bao, Daichi Zhang, Hansong Zhang, Yong Li, Yutao Yue, Shiming Ge · 2026
Read CD 2 : Constrained Dataset Distillation for Few-Shot Class-Incremental Learning by Kexin Bao, Daichi Zhang, Hansong Zhang, Yong Li, Yutao Yue, Shiming Ge (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08519.txt.
Introduction
Kexin Bao and five co-authors present CD2, a constrained dataset distillation method for incremental learning. Their 2026 arXiv paper targets few-shot class-incremental learning, where classes arrive with few training samples. Catastrophic forgetting is the key problem: new learning erases old classes.
Many methods store previous knowledge in external memory and treat it equally with new classes. The authors argue that habit fails to preserve what is essential. Constrained distillation instead compresses past knowledge into a small distilled dataset with safeguards. A compact memory that keeps the right examples could let field-deployed classifiers grow without bloating storage or losing early classes.
Catalog record
| CREATOR | Kexin Bao, Daichi Zhang, Hansong Zhang, Yong Li, Yutao Yue, Shiming Ge |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08519
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CD 2 : Constrained Dataset Distillation for Few-Shot Class-Incremental Learning Kexin Bao, Daichi Zhang, Hansong Zhang, Yong Li, Yutao Yue, Shiming Ge Abstract Few-shot class-incremental learning (FSCIL) receives significant attention from the public to perform classification continuously with a few training samples, which suffers from the key catastrophic forgetting problem. Existing methods usually employ an external memory to store previous knowledge and treat it with incremental classes equally, which cannot properly preserve previous essential knowledge. To solve this problem and inspired by recent distillation works on knowledge transfer, we propose a framework termed C onstrained D ataset D istillation ( CD 2 ) to facilitate FSCIL, which includes a dataset distillation module ( DDM ) and a distillation constraint module ( DCM ). Specifically, the DDM synthesizes highly condensed samples guided by the classifier, forcing the model to learn compacted essential class-related clues from a few incremental samples. The DCM introduces a designed loss to constrain the previously learned class distribution, which can preserve distilled knowledge more sufficiently. Extensive experiments on three public datasets show the superiority of our method against other state-of-the-art competitors. 1 Introduction In real-world applications such as robotics, healthcare, and remote sensing, dealing with sequential data streams and few-shot data arise frequently and often simultaneously 35 ; 22 . While deep neural models demonstrate impressive performance in static and data-abundant settings 32 ; 40 , it is still significantly challenging for them to learn new concepts continually with a very restricted quantity of labeled samples. In this situation, few-shot class-incremental learning (FSCIL) has emerged as a promising solution and has attracted much research attention 37 . In FSCIL, the model acquires extensive knowledge with sufficient labeled samples in the base session, and continually learns new knowledge from a few samples while retaining previously learned concepts in subsequent incremental sessions. Within FSCIL, the primary challenge lies in preventing catastrophic forgetting 14 ; 3 , a phenomenon where models tend to overwrite previously acquired concepts when confronted with new data due to the unavailability of data from previous sessions, especially learning with limited data. Figure 1: Red lines are the gap between actual extracted features and ideal features with all critical knowledge. Green arrows are the covariate shift. (a) Previous methods build a memory that dilutes critical knowledge, and treat the memory as equal to the real data, causing the covariate shift. (b) Using dataset distillation, the method can build a memory with more critical knowledge, but also faces the challenge of the covariate shift. (c) Our method builds and uses a memory incorporating distillation, which gains more critical knowledge and reduces the covariate shift. Recently, mainstream methods tend to freeze the backbone and adopt an external memory as an auxiliary tool during incremental sessions to preserve previously learned knowledge 19 ; 41 ; 2 . The memory stores a small amount of data from previous sessions by replaying real samples 19 ; 53 , generating samples 24 ; 1 or calculating pseudo features 10 ; 12 ; 43 ; 23 , enabling quick recall of old knowledge in subsequent sessions. Compared to only freezing the backbone, using a memory significantly boosts performance in incremental sessions, which illustrates that the memory enhances the ability of the model to revisit old knowledge and improves its information discrimination capacity. By leveraging prior experience effectively, using the memory offers a robust solution to catastrophic forgetting in an FSCIL setting. However, we find that existing methods may still learn redundant features and cannot preserve the previous distribution in incremental sessions. As shown by red lines in Figure 1 (a), when building the memory, previous methods dilute critical class-related knowledge due to mixing discriminative and redundant knowledge, which hinders the recall of old knowledge in subsequent sessions. Recently, Dataset Distillation (DD) is proposed to obtain a few highly condensed and informative samples from a dataset 47 ; 45 , which aligns with our goal of learning essential class-related knowledge. Therefore, we introduce DD for building a memory to obtain critical knowledge and synthesize highly informative samples as shown in Figure 1 (b). Further, there is a distribution gap between data in the memory and the new dataset in incremental sessions 42 . Treating them equally prompts the model to focus on distribution differences, leading to the covariate shift of old classes as illustrated by green arrows in Figure 1 (a) and (b). Facing the gap, we incorporate the core idea of knowledge distillation (KD) 11 during training. KD transfers knowledge from a teacher model (usually a well-trained, high-performing model) to a student model. And we constrain the change of distribution across continual sessions meticulously by putting the previously trained model as the teacher for the current model, which can mitigate the covariate shift and catastrophic forgetting. Combined with the core idea of distillation, our method can extract and curate critical knowledge as shown in Figure 1 (c). Specifically, we propose a C onstrained D ataset D istillation framework CD 2 to support FSCIL, which contains a more refined memory strategy and a more effective retention method. Firstly, we analyze existing popular memory strategies, including sample replay (reusing past training samples during incremental learning) and prototype computing (calculating the mean of intermediate features for each class) Then, to retain more essential knowledge, we propose a dataset distillation module ( DDM ) inspired by DD 47 . DDM synthesizes samples for each class to condense critical class-related knowledge, which retains knowledge stably and prevents performance degradation. Further, we employ a distillation constraint module ( DCM ) to transfer knowledge in a stable and flexible manner. The DCM promotes features and structures of old classes in the memory to align seamlessly between previous and current sessions, thereby enabling the model to constrain the distribution gap and utilize the memory more precisely and efficiently. Our main contributions can be summarized as follows: • We propose a framework termed Constrained Dataset Distillation ( CD 2 ) to facilitate the FSCIL task. To the best of our knowledge, we are the first to introduce dataset distillation to FSCIL, which could effectively captures critical knowledge. • We propose a dataset distillation module ( DDM ) to build a memory with more critical knowledge and a distillation constraint module ( DCM ) to reduce the covariate shift and maintain the stability of knowledge transmission, ensuring effective leveraging of dataset distillation. • Extensive and comprehensive experiments on three benchmark datasets are conducted, where all results demonstrate the effectiveness of our framework. 2 Related works 2.1 Few-Shot Class-Incremental Learning Few-shot class-incremental learning (FSCIL) enables the model to incrementally learn new knowledge while effectively retaining old knowledge with a few new samples 41 ; 2 . Some early methods update the whole model using the topology 37 and knowledge distillation 7 in incremental sessions, which suffer from catastrophic forgetting and over-fitting due to finetuning a large number of parameters. Based on this, mainstream methods finetune part of the parameters with the assistance of a memory during incremental learning, which can reduce overfitting while retaining certain old knowledge. Some methods select random 19 or adaptive 53 samples from old classes as the memory. However, replaying a few samples cannot obtain a representative and high generalization