Forget Less by Learning Together through Concept Consolidation

Arjun Ramesh Kaushik, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Venu Govindaraju · 2026

Read Forget Less by Learning Together through Concept Consolidation by Arjun Ramesh Kaushik, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Venu Govindaraju (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01963.txt.

Introduction

Personalized diffusion models that learn concepts one course at a time tend to forget the earlier ones. Arjun Ramesh Kaushik and four co-authors propose a different route. Their companion framework is called Forget Less by Learning Together. Concepts are learned together, not in sequence. Arrival is treated as concurrent and order-free.

A set-invariant module lets proxies guide feature selection across concepts, so knowledge transfers between them and old concepts are actively preserved while new ones arrive. Across three datasets the method improved concept retention by a clear margin. It averaged at least a two percent gain in image alignment scores. The tests covered ten incremental tasks. The results argue forgetting shrinks when concepts are allowed to teach each other.

Catalog record

CREATORArjun Ramesh Kaushik, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Venu Govindaraju
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Forget Less by Learning Together through Concept Consolidation Arjun Ramesh Kaushik, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Venu Govindaraju Abstract Custom Diffusion Models (CDMs) have gained significant attention due to their remarkable ability to personalize generative processes. However, existing CDMs suffer from catastrophic forgetting when continuously learning new concepts. Most prior works attempt to mitigate this issue under the sequential learning setting with a fixed order of concept inflow and neglect inter-concept interactions. In this paper, we propose a novel framework - Forget Less by Learning Together (FL2T) - that enables concurrent and order-agnostic concept learning while addressing catastrophic forgetting. Specifically, we introduce a set-invariant inter-concept learning module where proxies guide feature selection across concepts, facilitating improved knowledge retention and transfer. By leveraging inter-concept guidance, our approach preserves old concepts while efficiently incorporating new ones. Extensive experiments, across three datasets, demonstrates that our method significantly improves concept retention and mitigates catastrophic forgetting, highlighting the effectiveness of inter-concept catalytic behavior in incremental concept learning of ten tasks with at least 2% gain on average CLIP Image Alignment scores. Figure 1 : Forget Less by Learning Together (FL2T). Our method focuses on learning G concepts in an order-agnostic incremental learning problem setting with fewer parameters and fewer reference images while also mitigating catastrophic forgetting. Unlike previous works, we leverage inter-concept interactions positively. The above image showcases examples of generated images, with a source image and an associated text prompt (left column) as input. We evaluate the generated image (right column) against the SOTA model CIDM [ 5 ] (middle column). The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image. 1 Introduction Advancements in training, architectures, and datasets have enhanced text-conditioned generative models, enabling photorealistic text-to-image synthesis [ 31 , 34 , 3 ] . Conditional diffusion models facilitate high-fidelity image generation from text prompts and support user control over scene layout and sketches [ 24 , 29 , 6 , 43 ] . This level of quality in text-conditioned generative models opens up new avenues for customization and personalization, allowing for the generation of images featuring specific objects or individuals in novel contexts or backgrounds. The ability to precisely capture concepts from reference images and seamlessly integrate them with text prompts is a significant advancement. Early approaches to model customization typically involved fine-tuning a subset of model parameters [ 50 ] or text token embeddings [ 7 ] using a small set of user-provided reference images, often with additional regularization techniques [ 44 ] . However, this fine-tuning process, which is required for each new concept, is both time-intensive and computationally costly. As a result, recent efforts have shifted toward developing tuning-free methods that eliminate the need for per-object optimization during inference, thus offering a more efficient and scalable solution for customization. Concurrently, efforts are underway to expand the capability of model customization beyond a single, individual concept, aiming to seamlessly compose multiple new concepts together. However, this approach introduces additional challenges, such as the integration of unseen concepts and the potential for catastrophic forgetting of previously learned personalized concepts when new concepts are learned in succession through a concept-incremental framework. Additionally, issues like concept neglect may arise when performing multi-concept composition based on user-defined conditions [ 5 , 10 ] . A common setup in this area involves continuous customization with new, fine-grained concepts that have not been encountered before, typically with only a few examples provided for each new concept. This setup aligns with the concept of continual learning, which focuses on training a model across a sequence of tasks, each with a distinct data distribution, while striving to retain previously acquired knowledge without interference or forgetting. The continual learning framework provides a promising direction for multi-concept personalization [ 20 , 39 , 5 ] . Fundamentally, multi-concept personalization is closely related to training with larger datasets, as it inherently involves access to a greater number of reference images compared to single-concept personalization [ 5 , 12 , 36 ] . Existing approaches predominantly adopt a sequential concept learning paradigm, where new concepts are learned one after another. While this is practical, it raises the question: If reference images for all concepts are available from the outset, why not leverage them for simultaneous joint learning? This paper aims to address this critical gap by proposing a framework that enables the concurrent learning of multiple concepts, allowing each concept to aid in the learning of others [ 20 ] . Beyond mitigating catastrophic forgetting, this joint learning approach fosters mutual knowledge transfer between concepts, enhancing overall learning efficiency and improving the quality of multi-concept personalization. Most vision-language problems are instance-based, typically involving the mapping of a fixed-dimensional input tensor to a corresponding target value. However, multi-concept learning can be better formulated as a set-input problem [ 22 ] , where the input consists of a set of concepts or reference images, and the goal is to generate a corresponding output for the entire set. Effectively addressing this requires treating multi-concept learning within the framework of set-based modeling. A model designed for set-input problems must satisfy two key properties: (1) permutation invariance , meaning that the model’s output should remain unchanged regardless of the order in which input elements are presented, and (2) scalability , ensuring that the model can handle input sets of varying sizes without requiring retraining. Building upon these principles, this paper introduces a novel approach to multi-concept learning that jointly learns from multiple concepts while adhering to these fundamental properties. Specifically, the contributions of this paper are as follows: 1. We propose a two-step joint / parallel learning framework - Forget Less by Learning Together (FL2T) - using a text-to-image diffusion model, where we offer flexibility in the order of concept inflow. 2. Utilizing the permutation invariance property of transformers, we develop a concept interaction module that uses proxy embeddings to help alleviate catastrophic forgetting. Additionally, FL2T also accounts for scalability, showing improved performance over SOTA methods with fewer parameters and fewer reference images. 2 Related Work Continual learning for generative models. For VAEs, Ye, Bors [ 47 , 48 ] propose online cooperative memorization and continual generative knowledge distillation to mitigate forgetting. Within diffusion models, Zaj et al . [ 51 ] provide an early study that benchmarks Continual Learning baselines and reports timestep-dependent forgetting patterns and strong replay variants. Masip et al . [ 28 ] introduce generative distillation tailored to diffusion, improving stability across tasks. Recent works [ 49 ] also explore lifelong training settings beyond still images, e.g., lifelong video diffusion from a single stream using experience replay and dynamic architectural expansion for diffusion [ 46 ] . Forgetting in personalization/customization. Open-world customization emphasizes that adapting a model to new concepts can induce broad semantic and appearance drift, i.e., forgetting beyond the targeted concepts [ 21 ] . In continual customization, Continual Diffusion (C-LoRA) [ 39 ] adds concepts sequentially with parameter-efficient adapters and explicit self-regularization. Personalization and multi-concept customization. Foundational personalization methods such as Textual Inversion [ 7 ] and DreamBooth [ 36 ] adapt large text-to-image models to user-provided subjects from a few sample images. For multi-concept learning and composition, Multi-Concept

Read the complete text