Forget Less by Learning from Parents Through Hierarchical Relationships Arjun Ramesh Kaushik, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Nalini K. Ratha, Venu Govindaraju Abstract Custom Diffusion Models (CDMs) offer impressive capabilities for personalization in generative modeling, yet they remain vulnerable to catastrophic forgetting when learning new concepts sequentially. Existing approaches primarily focus on minimizing interference between concepts, often neglecting the potential for positive inter-concept interactions. In this work, we present Forget Less by Learning from Parents (FLLP), a novel framework that introduces a parent-child inter-concept learning mechanism in hyperbolic space to mitigate forgetting. By embedding concept representations within a Lorentzian manifold, naturally suited to modeling tree-like hierarchies, we define parent-child relationships in which previously learned concepts serve as guidance for adapting to new ones. Our method not only preserves prior knowledge but also supports continual integration of new concepts. We validate FLLP on three public datasets and one synthetic benchmark, showing consistent improvements in both robustness and generalization. Introduction Custom diffusion models (CDMs) ( 11 ; 13 ) have emerged as a powerful tool for personalizing image generation by adapting general-purpose text-to-image models to individual users or specific concepts. These models enable the synthesis of images from user-provided reference images, offering a pathway to generate personalized content. The growing interest in model customization reflects a broader goal of user-centric generative modeling, where models are tailored to represent distinct objects, styles, or identities that are not present in the training data. Text-conditioning enhances controllability and flexibility in diffusion models ( 26 ; 4 ) by guiding these models to generate semantically aligned visual content. Early personalization methods achieved this by fine-tuning a subset of model weights ( 37 ) or by learning new token embeddings ( 11 ) from a handful of user images. While effective, these methods necessitate per-concept optimization, resulting in increased computational overhead and significant time requirements. Recent work addresses these limitations through tuning-free methods ( 35 ) , without the need for re-training during inference. Beyond individual concept customization, recent research has focused on multi-concept personalization, composing multiple novel concepts into a single generation task ( 17 ; 32 ; 7 ) . Compared to single-concept settings, multi-concept customization entails diverse reference images and complex compositional prompts , leading to challenges such as handling previously unseen concepts, reducing interference among concepts, and maintaining prior knowledge across tasks. These challenges align naturally with the goals of continual learning, where a model learns incrementally from a sequence of concept-specific tasks without catastrophic forgetting. Recent mitigation strategies for catastrophic forgetting in CDMs, such as elastic weight consolidation, latent replay layers, and gradient matching techniques, primarily focus on isolating concepts through regularization and memory optimization. While these methods reduce immediate interference, they often fail to harness the latent relationships between learned concepts ( 14 ; 7 ) . Unlike current machine learning systems, human learning naturally excels at transferring knowledge across tasks, an ability rooted in our capacity to organize concepts into hierarchical structures. Research shows that we dynamically update them to support both long-term retention and generalization. Further, recent findings reveal that the brain itself is wired to accumulate and refine such hierarchical conceptual knowledge ( 34 ) . Motivated by these insights, we introduce Forget Less by Learning from Parents (FLLP) , the first framework to leverage inter-concept interactions positively, to the best of our knowledge. FLLP explicitly models hierarchical concept relationships to emulate human-like knowledge transfer. Like cognitive hierarchies, our approach projects concept embeddings into hyperbolic space ( 1 ) , a natural fit for encoding hierarchical structures due to its intrinsic tree-like geometry ( 6 ; 31 ) . Within this manifold, we enforce hierarchical parent entailment, requiring new concepts to reside within the entailment cones of their predecessors. This constraint mirrors how humans anchor novel skills (e.g., motorcycle control) to foundational schemas (bicycle balance), preserving conceptual dependencies while enabling adaptive reuse ( 34 ) . Our work makes three core contributions : (1) FLLP leverages natural hierarchies formed by hyperbolic embeddings to model inter-concept interactions. (2) We propose a Hierarchical Parent Entailment Loss using the union-find algorithm ( 9 ) to form a parent-chain that enforces new concepts to lie within the parent entailment cones. (3) We utilize this hierarchical learning to train CDMs continually and achieve SOTA results across three public datasets and one synthetic benchmark. To the best of our knowledge, this is the first contribution towards leveraging inter-concept interactions rather than mitigating them for a continual learning problem in CDMs. Figure 1: Continual Learning for Concept Personalization and Hyperbolic Geometry. (a) The Continual Learning (CL) paradigm involves training a model on a sequence of tasks or concepts (e.g., Bulbul, Zebra, Chow-chow) over time. A major challenge in this setup is Catastrophic Forgetting, where the model tends to lose knowledge of previously learned concepts as it acquires new ones. (b) In Euclidean space, all concepts coexist without any inherent hierarchy. In contrast, hyperbolic geometry enables hierarchical representation of concepts. Importantly, hyperbolic distances ( blue line ) are fundamentally different from Euclidean distances ( green line ). Related Work Incremental Concept Learning. Recent class-incremental learning methods focus on mitigating catastrophic forgetting via regularization, distillation, and rehearsal, but are largely developed for discriminative classifiers with fixed label spaces rather than generative models with continuous concept manifolds ( 38 ) . The survey in ( 38 ) emphasizes that interference between related classes remains a central limitation even under strong regularization or memory mechanisms. In text-to-image diffusion, lifelong and personalized approaches such as L2DM ( 33 ) , C-LoRA ( 32 ) , and ConceptGuard ( 14 ) adapt these ideas to the noise-prediction space by attaching concept-aware adapters and distillation losses to preserve old concepts during continual customization. However, they still treat each concept as an essentially independent unit, offering only coarse control over intra-concept variation and higher-order concept relationships. Intra-Concept Learning Dynamics. Multi-concept customization work shows that concept-specific updates interact in highly non-linear ways. Multi-Concept Customization, Mix-of-Show, and Multi-LoRA all report that composing several learned concepts or LoRA adapters can lead to strong interference and style leakage, especially when concepts are visually similar ( 17 ; 13 ; 41 ) . Cross-Attention Bootstrapping further reveals that cross-attention heads in diffusion models specialize to distinct object and attribute factors, and that misaligned heads are a key source of spurious edits and concept entanglement ( 40 ) . Together, these results motivate architectures that explicitly separate and recombine sub-concepts rather than treating each personalized token or adapter as monolithic. Catastrophic Forgetting in Concept Diffusion. Within text-to-image diffusion specifically, forgetting and confusion arise because concept information is distributed across the UNet, cross-attention layers, and token embeddings and is refined across timesteps. L2DM and C-LoRA retain concepts by distilling the denoising trajectory of earlier models and routing gradients through concept-specific low-rank adapters ( 33 ; 32 ) , while ConceptGuard disentangles forgetting from confusion and shows that overlapping attention patterns between related concepts are a dominant failure mode ( 14 ) . These systems markedly reduce catastrophic forgetting, but their parameter cost grows roughly linearly with the number of concepts and they still provide only implicit control over inter- and intra-concept interactions. Hyperbolic Geometry. Standard latent diffusion models operate in Euclidean latent spaces ( 29 ) , which are poorly suited for representing deep hierarchies and tree-like concept structure. Hyperbolic vision-language models such demonstrate that negative curvature and exponential volume growth enable compact, hierarchy-aware image-text embeddings with improved semantic organization ( 6 ) . HypDiff ( 10 ) proposes a hyperbolic latent diffusion model for graph generation, so as to preserve topological properties such as hierarchy and community structure during generation. However, these methods largely treat hyperbolic geometry as a static representation choice and do not address continual adaptation or concept interference in text-to-image diffusion. Limitations of Current Approaches. Across these lines of work, three gaps remain: (1) continual diffusion methods prioritize per-concept preservation but model concepts as independent, limiting intra-concept specialization and compositional recombination ( 33 ; 32 ; 14 ) ; (2) multi-concept customization exposes strong interference between learned edits without providing a principled latent structure for resolving it ( 17 ; 13 ; 41 ) ; and (3) hyperbolic generative models exploit hierarchical geometry but are not designed for dynamic, concept-aware updates over time ( 6 ; 21 ) . FLLP targets this intersection by coupling concept-aware continual updates with an explicitly structured latent space that is designed to accommodate evolving concept hierarchies. Problem Definition We consider the problem setting introduced in CIFC ( 7 ) , which considers a sequential inflow of concepts. The model learns from an undefined series of text-guided concept customization tasks T=\{T_{c}\}_{c=1}^{C} , where C denotes the total number of tasks. Each task T_{c} consists of a dataset T_{c}=\{(x_{c}^{k},p_{c}^{k},Y_{c})\}_{k=1}^{N} where N is the number of reference images in the task, x_{c}^{k} is an image, p_{c}^{k} is a text prompt (e.g., “photo of a [V cat ]”), and Y_{c} represents the concept space. The setting supports diverse customization tasks, including multi-concept generation ( 41 ) , style transfer ( 39 ) , and image editing ( 2 ) . There are three constraints in this problem - (1) All concepts are distinct: Y_{c}\cap\left(\bigcup_{i=1}^{C-1}Y_{i}\right)=\emptyset indicating that new concepts in task c are distinct from those learned in the other C-1 tasks. (2) Concepts are learned in the order of inflow: T=\{T_{1},T_{2},\dots,T_{C}\} , where the c -th concept T_{c} is learned with access only to prior concepts \{T_{1},\dots,T_{c-1}\} for c=1,\dots,C . This models a unidirectional learning flow, preventing access to future unseen concepts. (3) No-replay constraint: No memory storage is allocated to retain training images from past tasks, ensuring purely incremental learning of personalized concepts. This setting allows models to continuously adapt to new personalization tasks while being exposed to catastrophic forgetting of prior concepts. Understanding Catastrophic Forgetting Catastrophic forgetting refers to the tendency of neural networks to lose previously acquired knowledge when trained on new tasks. This phenomenon arises from the stability-plasticity dilemma, where neural networks must balance retaining existing information with adapting to new inputs ( 3 ) . Catastrophic forgetting occurs when parameter updates for a new task T_{\text{new}} overwrite critical weight configurations optimized for prior tasks T_{\text{old}} : P_{\text{old}}(W)\ll P_{\text{new}}(W) , where W represents network weights, and P denotes task performance. This work advances the central hypothesis that catastrophic forgetting can be fundamentally mitigated through the use of hierarchical knowledge transfer mechanisms. 1D Gaussian Setup To enable a systematic and interpretable analysis of catastrophic forgetting, we construct a synthetic one-dimensional dataset that serves as a controlled surrogate benchmark (see fig. 2 ). This benchmark comprises five sequential tasks, each defined by a Gaussian distribution with a distinct mean and unit variance, thereby emulating the concept drift characteristic of continual learning scenarios. A shared 1D UNet diffusion model is employed to reconstruct the data distribution associated with each task (see appendix for architectural details). To quantify forgetting, we compute the sum of absolute differences between the generated and true means across tasks. This serves as our primary evaluation metric, referred to as the forgetting rate , with lower values indicating improved retention of previously learned concepts. What Causes Catastrophic Forgetting? The catastrophic forgetting observed in the 1D Gaussian benchmark fig. 2 stems from two interrelated mechanisms: parameter interference and task-specific representation collapse . When training sequentially on tasks \mu_{i} , gradient updates for new tasks: \Delta W\propto\nabla_{W}\mathcal{L}_{\text{MSE}}(\mu_{\text{new}}) , perturb the network’s shared parameters W , overwriting features critical for reconstructing prior distributions \mu_{\text{old}} . This interference is amplified by the UNet’s frozen architecture, which forces all tasks to compete for representation space through a fixed set of weights. Parameter interference emerges from fundamental limitations in gradient-based optimization. When learning task \mu_{\text{new}} , the model computes weight updates proportional to the gradient of the current task’s MSE loss: \Delta W\propto-\eta\frac{\partial}{\partial W}\|Gen(\mu_{\text{new}})-\mu_{\text{new}}\|^{2} where Gen represents the generated output. These updates create vector fields in parameter space that optimally reduce \mu_{\text{new}} ’s reconstruction error but catastrophically displace solutions for previous tasks \mu_{\text{old}} . In our 1D setup, the frozen UNet architecture compounds this issue; though its core weights remain fixed, the task-specific token embeddings must encode all distributional shifts through a single projection layer. This creates a representational bottleneck where embedding vectors compete to map multiple Gaussian means ( \mu_{1},...,\mu_{5} ) through identical network transformations, leading to destructive interference patterns ( 23 ) . Task-specific representation collapse occurs when sequential training causes embeddings for distinct \mu_{i} to converge toward a shared subspace. The model’s token embeddings \{E(\mu_{i})\} gradually align with the dominant gradient directions from new tasks at t -th training step: E(\mu_{i})^{t+1}=E(\mu_{i})^{t}-\eta\sum_{j=1}^{k}\frac{\partial\mathcal{L}_{\text{MSE}}}{\partial E(\mu_{j})} . With no task-specific signals, the model optimizes for a single set of parameters to minimize the aggregate reconstruction error across all tasks. The mean of means emerges as the Nash equilibrium for minimizing the reconstruction error. This is demonstrated in the first row of Fig. 2 , where the model converges near the mean-of-means 9.6 , and achieves a forgetting rate of 18.6 . Analysis: Fig. 2 illustrates the visual outcome of our experiments. Each subplot compares the true and generated distributions for a given task after training on the final task, denoted in blue and red colors, respectively. The matrix layout of subplots is sorted in descending order of forgetting rate in different approaches. From top to bottom, we observe a monotonic reduction in error, indicating that the model increasingly preserves information about earlier concepts. The visual alignment of blue (true) and red (generated) bumps in lower rows highlights effective knowledge retention, a desirable trait for continual generative modeling. Notably, this alignment is most evident in Task 1, Task 2, and Task 3, where the extent of forgetting is also visualized by the length of the arrow indicating the mean shift. While we have not yet formally introduced our proposed method, FLLP, we offer a brief preview to anchor the reader’s attention. CIDM ( 7 ) achieves a forgetting rate of 13.2 , whereas our approach, FLLP, lowers this further to 11.4 . For context, the vanilla continual learning baseline yields a significantly higher forgetting rate of 18.6 . As a reminder, lower forgetting rates indicate better retention of previously learned concepts. 3 9 15 Continual Learning Task 1 3 9 15 Task 2 3 9 15 Task 3 3 9 15 Task 4 3 9 15 Task 5 3 9 15 CIDM 3 9 15 3 9 15 3 9 15 3 9 15 3 9 15 FLLP (Ours) 3 9 15 3 9 15 3 9 15 3 9 15 Figure 2: Quantitative Analysis of 1D Gaussian. The length of the orange arrows reflects the forgetting rate for each concept, with shorter arrows indicating better retention. CIDM ( 7 ) improves upon the baseline by achieving a lower forgetting rate of 13.2. FLLP (Ours) further reduces this to 11.4. The visibly shorter orange arrows in the bottom row corroborate the improved knowledge retention of FLLP over CIDM. (Best viewed by zooming in) Hyperbolic Trees Enable Optimal Grouping of Concepts In this section, we discuss the learning objectives of FLLP to model inter-concept interactions through hyperbolic representations of previously learned concepts and incoming concepts. We begin by introducing hyperbolic geometry and the tools required to implement our methodology. To adapt the \mathbf{c} -th customization task T_{c} , we employ LoRA ( 15 ; 36 ) to fine-tune the pretrained denoising UNet, commonly denoted using \epsilon . Background into Hyperbolic Geometry We adopt the Lorentz model of hyperbolic geometry for developing our framework. In this model, the n -dimensional hyperbolic space \mathbb{H}^{n} is realized as the upper sheet of a two-sheeted hyperboloid embedded in \mathbb{R}^{n+1} , endowed with the Lorentzian inner product (See Sec. A ). The model’s key advantage lies in its ability to express geodesics, distances, and projections in closed form, which is highly beneficial for optimization in high-dimensional settings. Following the study of special relativity theory ( 24 ) , the axis of symmetry of the hyperboloid is referred to as the time dimension , while the remaining n axes form the space dimensions . We adopt this terminology throughout, writing vectors in \mathbb{R}^{n+1} as [\mathbf{x}_{\text{space}},x_{\text{time}}] . We will now discuss a few tools from hyperbolic geometry to realize the learning objectives for our model. Geodesics: The shortest path between two points on a Riemannian manifold is defined as a geodesic . In the Lorentz model of hyperbolic space, geodesics are defined as the intersections between the hyperboloid and hyperplanes that pass through the origin of the ambient space \mathbb{R}^{n+1} . The Lorentzian distance between two points x,y\in\mathcal{L}^{n} , where k is the constant negative curvature, is given by d_{\mathcal{L}}(x,y)=\frac{1}{\sqrt{k}}\cosh^{-1}(-k\langle x,y\rangle_{\mathcal{L}}) . Aperture: The aperture defines the half-angle of the entailment cone originating from a point x in hyperbolic space, for a small constant K (typically set to 0.1) is defined as \texttt{Aper}(x)=\sin^{-1}\left(\frac{2K\sqrt{k}}{\|x_{\text{space}}\|}\right) . Exterior Angle: The exterior angle between two Lorentzian embeddings x and y measures how far y lies from the central axis of x ’s entailment cone. This is defined as the angle between x and y relative to the origin O , subtracted from \pi , \texttt{Ext}(x,y)=\cos^{-1}\left(\frac{y_{\text{time}}+x_{\text{time}}k\langle x,y\rangle_{\mathcal{L}}}{\|x_{\text{space}}\|\sqrt{(k\langle x,y\rangle_{\mathcal{L}})^{2}-1}}\right) Continual Learning in Diffusion Models using Hyperbolic Geometry We present a novel learning objective - Hyperbolic Parent Entailment Loss - to model hierarchical relationships between concepts using hyperbolic geometry in Algorithm 1 (Appendix). This loss function is governed by three crucial design choices: (1) transferring embeddings from Euclidean space to Lorentz hyperboloid, (2) finding the parent concept(s) for the incoming concept, and (3) designing a training objective to ensure the child concept (incoming concept) is learned through parent guidance (previously learned concepts). Transferring embeddings to hyperboloid: Let an embedding vector be \mathbf{v}\in\mathbb{R}^{n} . We aim to apply a transformation such that the resulting vector \mathbf{x} lies on the Lorentz hyperboloid \mathcal{L}^{n}\subset\mathbb{R}^{n+1} . Let the vector \mathbf{v}=[\mathbf{v},0]\in\mathbb{R}^{n+1} . We observe that \mathbf{v} belongs to the tangent space at the origin \mathcal{O} of the hyperboloid, since the Lorentzian inner product satisfies: \langle\mathcal{O},\mathbf{v}\rangle_{\mathcal{L}}=0. Thus, we parameterize only the space components of the Lorentz model (i.e., \mathbf{v}=\mathbf{v}_{\text{space}} ). This parameterization simplifies the exponential map. Instead of computing the full map in \mathbb{R}^{n+1} , we write the exponential map using only the space components: \mathbf{x}_{\text{space}}=\frac{\sinh(\sqrt{c} \, \|\mathbf{v}_{\text{space}}\|)}{\sqrt{c}\,\|\mathbf{v}_{\text{space}}\|}\cdot\mathbf{v}_{\text{space}} . Since the first term is zero, and the Lorentzian norm of \mathbf{v} simplifies to the Euclidean norm of the space components: \|\mathbf{v}\|_{\mathcal{L}}^{2}=\langle\mathbf{v},\mathbf{v}\rangle_{\mathcal{L}}=\langle\mathbf{v}_{\text{space}},\mathbf{v}_{\text{space}}\rangle=\|\mathbf{v}_{\text{space}}\|^{2}, The corresponding time component x_{\text{time}} can then be computed from \mathbf{x}_{\text{space}} as shown in Sec. A . The resulting vector \mathbf{x}=[x_{\text{time}},\mathbf{x}_{\text{space}}] lies on the Lorentz hyperboloid, eliminating the need for an orthogonal projection step. This parameterization is similar to ( 6 ) and simpler than prior approaches that operate in the full ambient space \mathbb{R}^{n+1} ( 18 ; 19 ) . Figure 3: Qualitative Analysis. We compare the generated images in TI ( 8 ) , CIDM ( 7 ) and FLLP (Ours) on CelebA ( 22 ) and CIFC ( 7 ) . The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image. The hyperbolic tree indicates the parent chain of concepts that the model traversed in learning the incoming concept. Parent Concept Identification: Let the CLIP ( 28 ) image features of the c -th concept be v_{c}\in\mathbb{R}^{s\times n} , where s is the number of source images and n is the feature dimensionality. Let t_{c}\in\mathbb{R}^{(c-1)\times\text{len}(Y_{c})\times n} represent the learned token embeddings corresponding to c-1 previously known concepts, where \text{len}(Y_{c}) corresponds to the number of tokens. The image features and learned token embeddings are averaged to obtain \bar{v_{c}}\in\mathbb{R}^{1\times n} and \bar{t_{c}}\in\mathbb{R}^{(c-1)\times n} . We define a recursive search strategy to construct a parent chain for the new concept, denoted by \mathcal{P_{\text{new}}} , where each parent will serve as an anchor. The intuitive idea is to iteratively move up the tree from the leaf node to root-level abstractions, as detailed in the Algorithm. 2 (Appendix). Let Z_{\text{img}} and Z_{\text{old}} denote the hyperbolic projections of \bar{v_{c}} and \bar{t_{c}} , respectively. We initialize the current embedding \mathbf{z}_{\text{curr}}\leftarrow Z_{\text{img}} and construct a search pool \mathcal{S}=Z_{\text{old}}\cup\{Z_{\text{img}}\} , indexed by \mathcal{I}=\{0,\dots,c{-}1\}\cup\{-1\} , where -1 refers to the c -th concept. At each iteration, we compute the negative Lorentzian geodesic distance between \mathbf{z}_{\text{curr}} and all embeddings in \mathcal{S} , selecting the second-closest index (skipping the trivial self-match) as the parent candidate. If the parent index is already visited or corresponds to the new concept itself, we detect a loop and terminate. Otherwise, the selected parent is assigned to \mathcal{P_{\text{new}}} , and \mathbf{z}_{\text{curr}} is updated accordingly. This process recursively builds a chain of semantically relevant parent concepts, revealing the new concept’s hierarchical alignment with seen concepts in hyperbolic space. Note that the Lorentzian geodesic distance d_{\mathcal{L}} captures the exponential separation of nodes from the origin. Storage of Image Attention Maps: For each of the c-1 previously observed image attention maps, denoted by I_{\alpha} for \alpha\in\{1,\dots,c-1\} , we store their timestep-weighted mean representation. This is computed as: \bar{I}_{\alpha}=\sum_{ts=1}^{\text{timesteps}}\frac{1}{\text{ts}}I_{\alpha}^{(ts)} , where I_{\alpha}^{(ts)} denotes the attention map at timestep ts . This aggregation provides a more stable and representative summary of each concept’s attention behavior over time. Note that this does not violate the “no memory” constraint, as the constraint is aimed at restricting the storage of reference images of previous concepts. Inducing Hierarchical Structure Between Parent-Child Concepts: In hyperbolic space, an entailment cone C_{u} at a point u represents the region formed by a hyperbolic cone ( 12 ; 19 ) . These cones narrow as the point u moves farther from the origin. The half-aperture of C_{u} is defined in Eq. Geodesics: . Let x and y represent the hyperbolic projections (spatial and time components) of the image attention maps corresponding to a child and parent concept, respectively. The associated time components are computed separately, as shown in Sec. A . Inspired by prior work on hierarchical embeddings ( 12 ) , we utilize entailment cones to enforce a partial ordering between the image attention maps of concepts. In particular, we constrain the child concept x to lie within the entailment cone defined by its parent y . To encourage this behavior during training, we introduce a loss term that penalizes violations of this constraint. Specifically, we impose a penalty when the exterior angle between x and y exceeds the aperture of y , and \beta is a hyperparameter that controls the flexibility of the constraint: \mathcal{L}_{\text{entailParent}}=\max\left(0,\texttt{ext}(y,x)-\beta*\texttt{aper}(y)\right) (1) Training Following CIDM ( 7 ) , we capture semantic patterns that persist in different tasks through direct supervision of LoRA weights. Specifically, a layer-wise common subspace \mathbf{W}_{*}^{l} is introduced to share across tasks. Additionally, a learnable projection matrix \mathbf{H}_{i}^{l} encodes common semantic attributes. Together, we optimize by minimizing the reconstruction loss: \mathcal{L}_{1}=\sum_{i=1}^{g}\sum_{l=1}^{L}\left\|\Delta\mathbf{W}_{i}^{l}-\mathbf{H}_{i}^{l}\mathbf{W}_{*}^{l}\right\|_{F}^{2} . Finally, we modify the total loss function for the Custom Diffusion model (Refer Sec. A in appendix) with \mathbf{m}_{c}=\{\mathbf{m}_{c}^{l}\}_{l=1}^{L} representing layer-wise textual embeddings, and hyperparameter \gamma . Our proposed framework, FLLP, is trained on \mathcal{L_{\text{total}}} utilizing a dual-step optimization strategy ( 7 ) . \mathcal{L}_{\text{total}}=\mathbb{E}_{\begin{subarray}{c}z\sim\Phi(x_{c}^{k}),m_{c}\\ \epsilon\sim\mathcal{N}(0,I),t\end{subarray}}\left[\|\epsilon-\epsilon_{\theta^{\prime}_{c}}(z_{t}|m_{c},t)\|_{2}^{2}+\gamma_{1}\mathcal{L}_{\begin{subarray}{c}\text{entail}\\ \text{Parent}\end{subarray}}+\gamma_{2}\mathcal{L}_{1}\right] (2) 0 5 10 15 20 25 30 35 60 70 80 90 Number of concepts CLIP Score (a) Scalability Analysis CIDM (IA) CIDM (TA) FLLP (IA) FLLP (TA) 0 5 10 60 70 80 Number of concepts (b) LoRA vs. Img Attn Maps LoRA (IA) LoRA (TA) IAM (IA) IAM (TA) 0 10 20 30 0 0.1 0.2 Number of concepts Relative Frobenius Norm (c) Parameter Drift CIDM FLLP Figure 4: Ablation Studies. (a) FLLP scales to 35 concepts (limited by CLIP tokenizer) and consistently outperforms CIDM ( 7 ) on both IA and TA metrics. (b) Hyperbolic Parent Entailment Loss on LoRA weights benefits IA scores, but hurts TA scores. Thereby making Image Attention Maps (IAM) better suited for the loss function. (c) FLLP exhibits 22% lower parameter drift than CIDM, indicating the utilization of inter-concept interactions positively. Experiments To rigorously demonstrate the robustness and generalizability of our proposed model, we conduct comprehensive experiments across three diverse public datasets, evaluating performance using CLIP ( 28 ) Image Alignment (IA) and Text Alignment (TA) metrics. Each class is treated as a distinct concept in our continual learning setup. Instead of leveraging all available classes, we intentionally select a subset of ten to simulate the sequential arrival of new concepts, an essential characteristic of realistic continual learning. This controlled design is critical for isolating and faithfully evaluating the effects of catastrophic forgetting under meaningful, yet analytically tractable, conditions. Remark (Scope of Concept Evaluation): In this work, we focus our analysis on a fixed set of 10 concepts across all experiments. This choice is informed by the architectural constraint imposed by the CLIP tokenizer, which allows only 77 learnable tokens ( 28 ) . Since each concept typically consumes 2–3 tokens, the upper bound for feasible training is approximately 35 concepts. While we do present an ablation study extending beyond 10 concepts to examine scalability, our core evaluation remains grounded in 10 concepts, a practical and representative regime for studying catastrophic forgetting. Importantly, our methodology does not inherently limit extension to larger concept sets. Implementation and dataset details are provided in the Appendix. Quantitative and Qualitative Analysis Methods C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Avg. IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA Previous frameworks Finetuning 77.6 64.4 82.2 74.6 79.0 69.4 77.6 68.6 79.6 75.0 62.9 70.0 71.5 76.7 53.7 69.2 81.4 65.4 72.1 67.2 73.7 70.0 EWC ( 16 ) 78.7 67.1 83.8 77.5 80.4 72.7 80.3 77.9 80.7 76.7 64.0 72.3 76.5 74.2 57.1 72.0 84.4 66.0 73.1 70.4 75.9 72.7 LWF ( 20 ) 80.4 70.8 79.7 75.2 80.9 71.0 77.4 77.4 80.9 76.0 61.8 71.7 73.2 76.3 53.5 72.9 78.1 72.5 74.7 70.0 74.1 73.4 CLoRA ( 32 ) 83.2 69.4 83.4 78.0 81.1 74.1 80.6 78.8 84.9 76.4 66.3 69.6 76.2 76.7 58.1 73.9 83.0 69.0 72.1 71.8 76.9 73.6 L2DM ( 33 ) 78.7 68.6 86.3 79.5 76.6 70.1 80.7 73.0 86.8 76.7 70.8 67.7 70.0 75.9 59.3 74.1 77.7 71.8 74.1 69.4 76.1 72.7 TI ( 8 ) 64.9 71.8 56.8 59.6 64.3 70.2 63.1 73.1 61.4 66.1 58.3 58.2 65.7 66.1 57.0 57.4 58.7 69.7 64.2 68.8 61.4 66.1 Comparison against SOTA framework CIDM ( 7 ) 83.6 75.3 86.4 78.1 82.9 74.0 80.8 81.1 86.5 78.2 69.5 70.1 73.7 74.7 56.9 74.3 82.4 73.5 75.9 70.2 78.0 74.8 FLLP (Ours) 83.9 75.7 87.0 82.3 85.8 72.7 83.5 82.7 87.6 76.9 72.2 72.1 84.2 72.5 58.8 77.5 80.2 77.3 76.4 71.6 80.0 76.1 CIFC dataset \Delta +0.3 +0.4 +0.6 +4.2 +2.9 -1.3 +2.7 +1.6 +1.1 -1.3 +2.7 +2.0 +10.5 -2.2 +1.9 +3.2 -2.2 +3.8 +0.5 +1.4 +2.0 +1.3 Previous frameworks TI ( 8 ) 44.3 62.4 47.2 65.2 49.6 61.5 52.5 57.4 50.2 58.7 49.8 54.7 51.0 53.7 51.3 56.4 41.7 61.9 50.0 65.3 48.8 59.7 Comparison against SOTA framework CIDM ( 7 ) 74.2 59.0 67.6 60.4 73.0 59.4 75.4 58.2 70.1 58.0 78.1 59.0 81.0 58.4 70.0 58.0 70.1 62.1 73.0 55.4 73.3 58.8 FLLP (Ours) 77.4 58.9 74.7 61.7 73.4 60.8 78.7 63.0 79.0 60.0 82.4 60.5 84.7 61.5 75.8 62.2 75.4 61.7 75.6 57.7 77.7 60.8 CelebA dataset \Delta +3.2 -0.1 +7.1 +1.3 +0.4 +1.4 +3.3 +4.8 +8.9 +2.0 +4.3 +1.5 +3.7 +3.1 +5.8 +4.2 +5.3 -0.4 +2.6 +2.3 +4.4 +2.0 Previous frameworks TI ( 8 ) 62.2 73.4 66.7 76.5 62.0 62.4 64.0 67.2 64.1 66.1 67.3 76.0 63.1 63.6 61.8 78.2 65.8 76.5 66.2 72.7 64.4 71.3 Comparison against SOTA framework CIDM ( 7 ) 80.8 74.1 71.4 77.9 81.3 81.7 83.0 82.6 79.2 83.7 84.5 69.6 76.7 79.3 87.5 73.7 82.2 79.9 85.0 82.8 81.2 78.5 FLLP (Ours) 82.1 73.9 73.4 78.9 82.6 82.5 83.1 84.0 79.7 85.1 82.2 69.7 75.5 79.2 90.0 75.0 86.4 79.8 87.5 80.9 82.3 79.0 ImageNet dataset \Delta +1.3 -0.2 +2.0 +1.0 +1.3 +0.8 +0.1 +1.4 +0.5 +1.4 -2.3 +0.1 -1.2 -0.1 +2.5 +1.3 +4.4 -0.1 +2.5 -1.9 +1.1 +0.5 Table 1: Quantitative Analysis. We compare our method FLLP against previous frameworks and the SOTA framework CIDM ( 7 ) across three datasets - CIFC ( 7 ) , CelebA ( 22 ) and ImageNet ( 5 ) . We observe an improvement in both Image Alignment (IA) and Text Alignment (TA) scores in all datasets. The higher IA score has been denoted in bold , and the higher TA value has been underlined . CIFC Dataset: As shown in Tab. 1 , we achieve superior IA and TA scores compared to CIDM ( 7 ) in 9 out of 10 concepts. Notably, FLLP achieved +9.2 IA scores on the C7 (Dog2) and +3.2 , being the largest TA gain on the C9 (Cat2). Over the entire dataset, FLLP demonstrates a gain in both IA and TA scores of +1.2 and 1.1 , respectively. Note that each concept is trained with a different value of \beta (Threshold parameter). CelebA Dataset: On the CelebA dataset ( 22 ) , FLLP surpasses CIDM with a notable gain of +4.4 in Image Alignment (IA) and +2.0 in Text Alignment (TA) scores. Among individual concepts, Person 5 (C5) exhibits the highest IA improvement with a margin of +8.9 , while Person 8 (C8) shows the most significant TA boost, achieving +4.2 over CIDM. ImageNet Dataset: On the third benchmark dataset, FLLP continues to outperform CIDM, achieving gains of +1.1 in IA and +0.5 in TA scores. Performance improvements are observed consistently across most concepts, with C9 (Wood Rabbit) exhibiting the largest IA gain of +4.4 . FLLP consistently outperforms CIDM across CIFC, CelebA, and ImageNet, showing notable gains in both IA and TA metrics. Qualitative Analysis: We provide a qualitative comparison of the synthesized outputs generated by three approaches in Fig. 3 : TI ( 8 ) , CIDM ( 7 ) , and our proposed FLLP framework, spanning CelebA ( 22 ) (top row) and CIFC ( 7 ) (bottom row) datasets. Each image is generated using a source image along with its corresponding text prompt. Colored arrows are overlaid to highlight perceptual differences: green arrows denote regions where the output exhibits desirable qualities such as identity preservation or faithful prompt alignment; red arrows highlight deficiencies such as concept drift, artifact introduction, or semantic inconsistency. Commonly, we observe identity mismatches and semantic misalignment across both datasets in previous frameworks that FLLP corrects. Additionally, we visualize the hyperbolic parent-child structure that FLLP leverages during training through a hyperbolic tree. These trees represent the conceptual trajectory taken by the model when incorporating a new concept. For example, in the case of C7 dog (from the CIFC dataset), FLLP uses a hierarchical parent chain consisting of Cat1 (C3), Rubber Duck (C2), and Painting (C6) to guide the learning of the new concept. In other words, concepts C2, C3, C6 and C7 form a group. Additional examples have been provided in the Appendix. FLLP enhances output quality by correcting identity and semantic errors through hierarchical hyperbolic guidance. Ablation Studies Why image attention maps instead of LoRA weights? Direct hyperbolic optimization on LoRA weights forces a simultaneous geometric transformation of both modalities. Image-related weights benefit, while text-related weights suffer ( 27 ) , resulting in a zero-sum game as elucidated in Fig. 4 (b) and Table 3 (Appendix). On the effect of \beta : As illustrated in Fig. 6 , 7 , 8 (Appendix) across the three datasets, each concept exhibits a distinct optimal threshold value. The threshold modulates the extent of parental influence on child concepts; lower values enforce stronger guidance, while higher values relax this constraint. Scalability: To evaluate FLLP beyond ten concepts, we conduct experiments on the CustomConcept101 dataset ( 17 ) . Due to CLIP’s tokenizer constraint allowing only 77 additional tokens ( 28 ) , we are limited to training on 35 concepts (considering 2–3 tokens per concept). As shown in Fig. 4 (a) and Table 4 (Appendix), FLLP outperforms CIDM by +2.1 on average IA scores and +1.0 on average TA scores. Comparing parameter drifts: We compare the evolution of LoRA weights in FLLP (Ours) and CIDM ( 7 ) using the Frobenius norm averaged across all layers on 35 concepts from the CustomConcept101 ( 17 ) dataset. As in Fig. 4 (c), FLLP exhibits 22% lower parameter drift, highlighting positive inter-concept interactions. Conclusion In this work, we introduced FLLP, a novel framework that leverages hyperbolic geometry and parent-child relational priors to alleviate catastrophic forgetting in CDMs. Unlike existing approaches that reduce inter-concept interactions, FLLP exploits structured inter-concept dependencies through a Lorentzian manifold, enabling positive transfer and robust continual adaptation. Across diverse benchmarks, our empirical results demonstrate that FLLP reliably enhances prior-knowledge retention, strengthens generalization, and scales effectively to new concepts. Appendix A Technical Appendices and Supplementary Material Inference Similar to CIDM, our inference strategy relies on Elastic Weight Aggregation (EWA). The learned LoRA weights are aggregated based on the semantic relationship between the stored concept token embedding ( \hat{c} ) and the current task’s prompt embedding ( \hat{e} ) to reduce forgetting. The aggregated low-rank weight \Delta\mathbf{\hat{W}}^{l} in the l -th transformer layer is formulated as: \Delta\mathbf{\hat{W}}^{l}=\sum_{i=1}^{c}\Delta\mathbf{W}_{i}^{l}\cdot\psi(\mathbf{\mathcal{M}})_{i} , and \mathbf{\mathcal{M}}=\max(\mathbf{\hat{c}}^{l}\cdot(\mathbf{\hat{e}}^{l})^{\top})\quad , where \max(\cdot) operates along the row axis, and \psi(\mathbf{\mathcal{M}}) denotes the frobenius-norm of \mathcal{M} . After aggregating the learned low-rank weights, we obtain the updated denoising UNet \epsilon_{\theta^{\prime}_{\ast}}(\cdot) for inference, where \theta^{\prime}_{\ast}=\theta_{0}+\Delta\theta_{\ast} and \Delta\theta_{\ast}=\{\Delta\mathbf{\hat{W}}^{l}\}_{l=1}^{L} . This formulation effectively consolidates the essential characteristics of all personalized concepts. Threshold Affects the Hierarchical Concept Tree In this section, we examine the influence of the threshold parameter \beta on the performance of our framework. As illustrated in Figures 6 , 7 , and 8 across the three datasets, each concept exhibits a distinct optimal threshold value. The threshold modulates the extent of parental influence exerted on child concepts within the hierarchy: lower values of \beta enforce stronger guidance, while higher values relax this constraint. Although the loss function is applied exclusively to the image attention maps, the concept token embeddings are indirectly shaped by the threshold, consequently altering the structure of the induced concept tree. This behavior is further substantiated in Table 5 , which demonstrates that varying the threshold results in different parent-child relationships within the concept hierarchy. Thresholding effectively filters which parent embeddings influence the child, allowing us to balance between inheritance and autonomy. This dynamic control is crucial in continual learning, where over-regularization can harm generalization, and under-regularization may lead to concept drift. Dataset t C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 0.1 {} {C1} {C2, C1} {C3, C2} {C3, C2} {C3, C2} {C6, C2, C3} {C6, C2, C3} {C6, C2, C3} {C6, C2, C3} 0.3 {} {C1} {C2, C1} {C3, C2} {C3, C2} {C3, C2} {C3, C2} {C3, C2} {C3, C2} {C3, C2} 0.5 {} {C1} {C2, C1} {C3, C2} {C3, C2} {C3, C2} {C6, C2, C3} {C3, C2} {C3, C2} {C3, C2} 0.7 {} {C1} {C2, C1} {C3, C2} {C3, C2} {C3, C2} {C6, C2, C3} {C6, C2, C3} {C6, C2, C3} {C6, C2, C3} 0.9 {} {C1} {C2, C1} {C3, C2} {C3, C2} {C3, C2} {C3, C2, C6} {C3, C2, C6} {C3, C2, C6} {C3, C2, C6} CIFC ( 7 ) 1.0 {} {C1} {C2, C1} {C3, C2} {C3, C2} {C3, C2} {C3, C2, C6} {C3, C2, C6} {C3, C2, C6} {C3, C2, C6} 0.1 {} {C1} {C1, C2} {C1, C2} {C1, C2} {C1, C5} {C1, C5} {C1, C5} {C1, C5} {C1, C5} 0.3 {} {C1} {C2, C1} {C2, C1, C3} {C2, C1, C4} {C2, C5, C1, C4} {C2, C6, C5} {C2, C6, C5} {C2, C6, C5} {C2, C6, C5} 0.5 {} {C1} {C1, C2} {C1, C3} {C1, C3} {C1, C3} {C1, C6, C2} {C1, C6, C2} {C5, C2, C6} {C1, C6, C2} 0.7 {} {C1} {C1, C2} {C1, C3} {C1, C3} {C1, C5} {C1, C6, C2} {C1, C6, C2} {C1, C6, C2} {C1, C6, C2} 0.9 {} {C1} {C1, C2} {C1, C2} {C1, C2} {C1, C2} {C1, C6, C2} {C1, C6, C2} {C1, C6, C2} {C1, C6, C2} CelebA ( 22 ) 1.0 {} {C1} {C1, C2} {C1, C3} {C1, C3} {C1, C3} {C1, C6, C3} {C1, C6, C3} {C1, C6, C3} {C1, C6, C3} 0.1 {} {C1} {C1, C2} {C2, C3} {C1, C2, C3} {C2, C3} {C2, C3} {C2, C3} {C2, C3} {C1, C2, C3} 0.3 {} {C1} {C1, C2} {C1, C2, C3} {C1, C2, C3} {C5, C3, C2} {C5, C3, C2} {C5, C3, C2} {C5, C3, C2} {C5, C3, C2} 0.5 {} {C1} {C1, C2} {C1, C2, C3} {C1, C2, C3} {C5, C3, C2} {C5, C3, C2} {C5, C3, C2} {C5, C3, C2} {C1, C2, C3} 0.7 {} {C1} {C2, C1} {C2, C3} {C4, C2, C3} {C4, C2, C3} {C4, C2, C3} {C5, C3, C2} {C5, C3, C2} {C2, C3} 0.9 {} {C1} {C1, C2} {C3, C2} {C3, C2} {C3, C2} {C5, C3, C2} {C5, C3, C2} {C3, C2} {C2, C3} ImageNet ( 5 ) 1.0 {} {C1} {C2, C1} {C2, C1} {C2, C3} {C2, C3} {C2, C3} {C2, C3} {C2, C3} {C2, C3} Table 2: Parent chains formed by each concept at different thresholds (t). Training the image attention maps at different threshold values also affects the token embeddings. Thereby, influencing the formation of the concept-tree. The parents have been ordered from nearest to farthest hyperbolic distance. The optimal parent chain for each concept has been denoted by bold-underline . 0 0.5 1 76 78 80 Threshold CLIP Score CIFC 0 0.5 1 60 65 70 75 80 Threshold CelebA 0 0.5 1 78 80 82 Threshold INet IA TA Figure 5: Average CLIP scores of all ten concepts. Effect of threshold parameter on the average CLIP Score across the three datasets. 0 0.5 1 75 80 85 CLIP Score C1 0 0.5 1 80 85 C2 0 0.5 1 70 75 80 85 C3 0 0.5 1 80 85 C4 0 0.5 1 75 80 85 90 C5 0 0.5 1 65 70 75 CLIP Score C6 0 0.5 1 70 75 80 85 C7 0 0.5 1 55 60 65 70 75 80 C8 0 0.5 1 75 80 C9 0 0.5 1 70 75 C10 IA TA Optimal Threshold Figure 6: Effect of threshold variation on the CIFC dataset. In this plot, we analyse the variation of CLIP scores ( IA and TA ) for different threshold values. We observe that each concept has a specific threshold value that exhibits maximum gains in scores. 0 0.5 1 60 65 70 75 CLIP Score C1 0 0.5 1 60 65 70 75 C2 0 0.5 1 60 65 70 75 C3 0 0.5 1 60 65 70 75 80 85 C4 0 0.5 1 55 60 65 70 75 80 C5 0 0.5 1 60 65 70 75 80 85 CLIP Score C6 0 0.5 1 60 65 70 75 80 85 C7 0 0.5 1 60 65 70 75 C8 0 0.5 1 55 60 65 70 75 80 C9 0 0.5 1 55 60 65 70 75 80 C10 IA TA Optimal Threshold Figure 7: Effect of threshold variation on the CelebA dataset. In this plot, we analyse the variation of CLIP scores ( IA and TA ) for different threshold values. We observe that each concept has a specific threshold value that exhibits maximum gains in scores. 0 0.5 1 75 80 85 CLIP Score C1 0 0.5 1 70 75 80 C2 0 0.5 1 80 85 C3 0 0.5 1 80 85 C4 0 0.5 1 75 80 85 C5 0 0.5 1 70 75 80 CLIP Score C6 0 0.5 1 70 75 80 C7 0 0.5 1 75 80 85 90 C8 0 0.5 1 80 85 C9 0 0.5 1 80 85 90 C10 IA TA Optimal Threshold Figure 8: Effect of threshold variation on the ImageNet dataset. In this plot, we analyse the variation of CLIP scores ( IA and TA ) for different threshold values. We observe that each concept has a specific threshold value that exhibits maximum gains in scores. Hyperbolic Constraint on Image Attention Maps vs. LoRA Weights In this section, we delve into the intuition behind our proposed approach and provide the reader with a compelling understanding of our results - quantitatively and qualitatively. Most prior works on custom diffusion models in a continual learning setup propose solutions by limiting the inter-concept interference. Notably, our first experiment, “CIDM with Cosine-guidance” on the CIFC dataset in Table 5 questions this strategy. We weight the Task Specific Loss ( \mathcal{L}_{1} ) based on the cosine-similarity scores between the token embeddings of the previous concepts and new concept, as shown in Eq. 3 . The comparable results of this experiment with “CIDM” alone prompted us to explore the idea of leveraging inter-concept interactions rather than reducing them. Intuitively, we arrived at the goal of transferring learning from one concept (parent) to another (child). \mathcal{L}_{1}=\sum_{i=1}^{g-1}\sum_{l=1}^{L}\lambda_{i}\mathbf{A}_{i}^{l}(\mathbf{A}_{g}^{l})^{\top} (3) Hyperbolic geometry allows us to exploit this potential relationship by naturally allowing embeddings to form tree-like hierarchies. This brings the question, why can’t we directly optimize on the LoRA parameters in the hyperbolic space? Direct hyperbolic optimization on LoRA weights forces a simultaneous geometric transformation of both modalities. Although image-related weights adapt well to hyperbolic hierarchy preservation, text-related weights will suffer from distorted similarity metrics, breaking original semantic relationships ( 27 ) . Thereby creating a zero-sum game where image gains come at text’s expense, as shown in Table 3 . To this end, we drive our focus on specifically only image attention maps of the incoming concept. Recall that our learning objective is designed to ensure that the attention maps of a child’s concept lie in the entailment cone of the parent. Methods C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Avg. IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA FLLP on LoRA weights 84.3 73.3 87.4 79.5 82.6 73.1 83.2 79.2 87.2 75.7 69.1 72.2 84.5 71.8 59.0 76.5 80.9 74.6 77.3 69.5 79.6 74.5 FLLP on image attention maps 83.9 75.7 87.0 82.3 85.8 72.7 83.5 82.7 87.6 76.9 72.2 72.1 84.2 72.5 58.8 77.5 80.2 77.3 76.4 71.6 80.0 76.1 \Delta -0.4 +2.4 -0.4 +2.8 +3.2 -0.4 +0.3 +3.5 +0.4 +1.2 +3.1 -0.1 -0.3 +0.7 -0.2 +1.0 -0.7 +2.7 -0.9 +2.1 +0.4 +1.6 Table 3: Hyperbolic constraint on LoRA weights instead of image attention maps. We compare the effects of Hyperbolic Parent Entailment Loss directly on LoRA weights and image attention maps. Experimental results indicate that FLLP performs better when hyperbolic constraints are applied on the image attention maps. The higher IA score has been denoted in bold , and the higher TA value has been underlined . Methods C1 - C5 C6 - C10 C11 - C15 C16 - C20 C21 - C25 C26 - C30 C31 - C35 Avg. IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA CIDM ( 7 ) 73.8 75.4 70.2 75.9 67.8 75.7 74.0 70.7 70.7 79.7 69.6 73.7 70.8 78.2 71.0 75.6 FLLP 76.1 76.5 71.7 77.2 69.5 76.4 75.9 71.4 73.3 81.0 71.9 74.7 73.2 78.7 73.1 76.6 \Delta +2.3 +1.1 +1.5 +1.3 +1.7 +0.7 +1.9 +0.7 +2.6 +1.3 +2.3 +1.0 +2.4 +0.5 +2.1 +1.0 Table 4: Scalability Analysis. To understand the robustness of FLLP on more than ten concepts, we consider 35 concepts (limited by CLIP tokenizer) from the CustomConcept101 dataset ( 17 ) . We observe that FLLP consistently outperforms CIDM across both IA and TA scores (averaged over five consecutive concepts). The higher IA score has been denoted in bold , and the higher TA value has been underlined . Methods C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Avg. IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA IA TA CIDM ( 7 ) 83.6 75.3 86.4 78.1 82.9 74.0 80.8 81.1 86.5 78.2 69.5 70.1 73.7 74.7 56.9 74.3 82.4 73.5 75.9 70.2 78.0 74.8 CIDM ( 7 ) with cosine-guidance 82.0 74.4 85.0 80.1 81.1 75.5 78.3 81.6 85.4 79.3 73.4 71.0 81.9 72.1 59.1 76.2 82.8 73.7 76.9 71.8 78.6 75.6 \Delta -1.6 -0.9 -1.4 +2.0 -1.8 +1.5 -2.5 +0.5 -1.1 +1.1 +3.9 +0.9 +8.2 -2.6 +2.2 +1.9 +0.4 +0.2 +1.0 +1.6 +0.6 +0.8 Table 5: Intuition behind leveraging inter-concept interactions. We compare the SOTA framework CIDM ( 7 ) against CIDM with cosine-guidance that uses Eq. 3 . The weighting is derived based on cosine-similarity scores of the token embeddings. We observe an improvement in both Image Alignment (IA) and Text Alignment (TA), validating our hypothesis that inter-concept interactions can be utilized positively. The higher IA score has been denoted in bold , and the higher TA value has been underlined . Computational Cost Analysis CIDM ( 7 ) and FLLP (Ours) exhibit comparable training times on the CIFC dataset - 4512s and 4910s, respectively. While hyperbolic operations in FLLP are inherently less efficient than their Euclidean counterparts, the overall training time is balanced since FLLP only considers a subset of prior concepts in its parent chain. Notably, inference time remains unchanged across both frameworks. Scalability Analysis To evaluate FLLP beyond ten concepts, we conduct experiments on the CustomConcept101 dataset ( 17 ) . FLLP builds upon CIDM ( 7 ) , which in turn uses the CLIP text encoder ( 28 ) . Due to CLIP’s tokenizer constraint allowing only 77 additional tokens ( 28 ) , we are limited to training on 35 concepts (considering 2–3 tokens per concept). Our results, as shown in Table 4 , indicate that FLLP scales well and consistently outperforms CIDM. Comparing Parameter Drifts To study the evolution of LoRA weights in FLLP (Ours) and CIDM ( 7 ) , we plot the Frobenius norm of the LoRA weights averaged across all layers against 35 concepts from the CustomConcept101 ( 17 ) dataset as shown in Fig. 9 . Both CIDM and FLLP show a clear trend in the Frobenius norm values of their LoRA parameters over the 35 concepts. Initially, both frameworks exhibit a rapid increase in their norm values, suggesting significant updates or adjustments to the parameters as learning progresses. For CIDM, the values start at 0.5 and climb to around 0.92-0.94 by step 19, where they stabilize with minor fluctuations. Similarly, FLLP begins at 0.47 and rises to 0.94-0.95 by step 11, stabilizing thereafter. This indicates that FLLP is able to incorporate concepts without significant parameter updates earlier than CIDM. Using the relative Frobenius norm as a measure, we calculated the average absolute difference between consecutive steps to quantify the parameter drift for each framework in Fig. 10 . For CIDM, the average drift is 0.018 . FLLP exhibits a 22% lower average drift at 0.014 , suggesting even less variation per concept on average compared to CIDM. Thereby exhibiting superior performance in addressing catastrophic forgetting. 0 10 20 30 0.5 0.6 0.7 0.8 0.9 1 Concept number Frobenius Norm CIDM (True) FLLP (True) Stability (FLLP) Stability (CIDM) Figure 9: Comparing True LoRA Frobenius Norms. Both frameworks exhibit an upward trajectory in their norm values, suggesting significant updates to the parameters with incoming concepts. FLLP is able to incorporate concepts without parameter updates earlier than CIDM, as evidenced by their stability points indicated by blue and red lines, respectively. 0 10 20 30 0 5\cdot 10^{-2} 0.1 0.15 0.2 0.25 Concept number Relative Frobenius Norm CIDM (Relative) FLLP (Relative) Figure 10: Comparing Relative LoRA Frobenius Norms. We compute the difference between LoRA Frobenius Norms as a metric to measure parameter drift. Based on this experiment, we observe that FLLP exhibits a 22% lower average parameter drift compared to CIDM, thereby showcasing its advantages in alleviating catastrophic forgetting. Understanding Catastrophic Forgetting in a 1D Synthetic Dataset Experimental Setup. The UNet is first trained on 1000 samples for \mu=1 and \sigma=1 . Subsequently, the UNet parameters are frozen, and only task-specific token embeddings are learned. During training, the model sequentially encounters 1000 samples for each task \mu_{i}=3i,\text{where }i\in[1,5] , mimicking a continual learning setting. After training on all tasks, we evaluate the model’s retention of past concepts by comparing the generated mean of the reconstructed distribution to its true mean. The sum of the absolute differences between the generated and true means acts as a quantitative indicator of forgetting, termed as forgetting rate : smaller differences suggest better retention and lower forgetting. We measure forgetting rate across three models using this setup: baseline continual learning, CIDM ( 7 ) , and FLLP (Ours). The baseline model is optimized only on Mean Squared Error (MSE), whereas CIDM uses a combination of MSE and Concept Consolidation Loss (CCL) ( 7 ) . We do not consider other loss functions since CCL is the point of difference in the learning objective between CIDM and FLLP (Ours). Hyperbolic Parent Entailment Loss Algorithm 1 Hierarchical Parent Entailment Loss X_{\text{new}} (image attention maps of new concept); X_{\text{old}} (image attention maps of previous concepts); F_{\text{img}} (CLIP image features of new concept); F_{\text{text}} (previous concepts’ token embeddings); t (aperture threshold) Z_{\text{new}},Z_{\text{old}}\leftarrow\texttt{expm}(X_{\text{new}}),\texttt{expm}(X_{\text{old}}) Z_{\text{img}},Z_{\text{text}}\leftarrow\texttt{expm}(F_{\text{img}}),\texttt{expm}(F_{\text{text}}) P\leftarrow\texttt{ParentSearch}(Z_{\text{img}},Z_{\text{text}}) \triangleright Selects parents by iteratively moving up the tree Loss\leftarrow[\ ] for each index i in P do Z_{p}\leftarrow Z_{\text{old}}[i] \theta\leftarrow\texttt{Ext}(Z_{p},Z_{\text{new}}) \triangleright Eq. Geodesics: \alpha\leftarrow\texttt{Aper}(Z_{p}) \triangleright Eq. Geodesics: Loss.\texttt{append}(\max(0,\theta-t\alpha)) end for return \texttt{mean}(Loss) To ensure hierarchical consistency in concept learning, we propose the Hyperbolic Parent Entailment Loss (Algorithm 1 ), which encourages new image attention maps to respect parent-child entailment structure. The algorithm operates by first exponentiating both the incoming concept’s image attention maps and the prior concepts’ maps to obtain their hyperbolic embeddings. These embeddings are then matched to their respective parent candidates using a tree traversal procedure ( ParentSearch ) guided by the CLIP-derived image and text feature spaces. For each identified parent, we extract the entailment angle \theta between the parent’s and the new concept’s embeddings, and weigh it against the parent’s aperture \alpha , its effective generalization radius. The loss penalizes violations of the entailment condition, i.e., when \theta exceeds the scaled aperture t\alpha , ensuring that children remain geometrically constrained within the conceptual cone of their parents. This loss formulation allows for selective, hierarchy-preserving updates to the embedding space without regressing previously learned semantic structure. ParentSearch Algorithm Algorithm 2 ParentSearch : Parent Chain Construction in Hyperbolic Space through Union-Find Algorithm 1: Z_{\text{img}} (new concept embedding); Z_{\text{old}} (old concepts’ embeddings) 2: \text{current\_emb}\leftarrow\text{new\_emb} 3: \text{parent\_chain},\text{visited}\leftarrow\emptyset,\emptyset 4: \text{visited}\leftarrow\emptyset 5: \text{search\_pool}\leftarrow\text{old\_embs}\cup\text{new\_emb} 6: \text{index\_list}\leftarrow[0,1,\dots,n-1]\cup[-1] 7: while True do 8: D\leftarrow-d_{\mathcal{L}}(\text{current\_emb},\text{search\_pool}) \triangleright Eq. 5 9: \text{top\_indices}\leftarrow\text{TopK}(D,\text{k=2},\texttt{min=True}) \triangleright Pick the smallest 2 10: \text{parent}\leftarrow\text{top\_indices}[1] \triangleright Since the smallest will be the embedding itself 11: \text{parent\_idx}\leftarrow\text{index\_list}[\text{parent}] 12: if (\text{parent\_idx}\in\text{visited})\lor(\text{parent\_idx}=-1) then \triangleright A loop is detected 13: break 14: end if 15: \text{visited}\leftarrow\text{visited}\cup\{\text{parent\_idx}\} 16: \text{parent\_chain}\leftarrow\text{parent\_chain}\cup\{\text{parent\_idx}\} 17: \text{current\_emb}\leftarrow\text{search\_pool}[\text{parent\_idx}] 18: end while 19: return parent_chain The ParentSearch algorithm (Algorithm 2 ) constructs a hierarchical parent chain for a given new concept by performing iterative nearest-neighbor traversal in hyperbolic space. Leveraging the geodesic distance metric d_{\mathcal{L}} in the Lorentz model, it identifies the closest ancestor candidates among all previously embedded concepts. To prevent cyclic dependencies and redundant loops, a union-find–style visitation set is maintained, ensuring that traversal proceeds only through unvisited nodes. At each step, the nearest neighbor (excluding the current embedding itself) is selected as the next parent, and its index is appended to the parent chain. This walk continues until a stopping criterion is met, either a previously visited node is encountered, or the traversal reaches the new embedding itself. The resulting chain serves as the foundation for enforcing entailment constraints, allowing each child to align itself within the conceptual envelope of its semantic predecessors. This mechanism elegantly encodes implicit tree structures without the need for explicit supervision or symbolic hierarchies. The algorithm has a worst-case time complexity of O(c-1) when training for the c -th concept. Experiments, Implementation Details and Evaluation Metrics To showcase the robustness and generalizability of our model, we have used three public datasets: CIFC ( 7 ) , CelebA ( 22 ) , and ImageNet ( 5 ) . Each dataset consists of 10 different concepts with 3-5 reference images for each task. We use Stable Diffusion (SD-1.5) ( 29 ) as the pretrained model and the CIDM ( 7 ) as the foundational framework for all experiments. The training is conducted on two Nvidia H100 GPUs, with a fixed initial learning rate of 1e^{-3} for updating textual embeddings and 1e^{-4} for optimizing the U-Net. The curvature of the hyperboloid is a learnable parameter initialized to 1.0 , and optimized with a learning rate of 1e-4 . Empirically, we set \gamma_{1}=\gamma_{2}=0.1 in Eq. 2 . Following our experiments under our problem setting as in Sec Problem Definition , we evaluate our generated images across 2 metrics - Image Alignment (IA) and Text Alignment (TA). Image Alignment (IA) scores are computed using the CLIP ( 28 ) image encoder, comparing the similarity of features between generated images and reference images. Similarly, we utilize the text encoder of CLIP ( 28 ) to evaluate the text-image similarity between the input prompt and synthesized image for the Text Alignment (TA) scores. To analyze the comparisons between our model and SOTA frameworks, we follow the evaluation strategy defined in ( 7 ) . Specifically, 20 evaluation prompts are introduced for each of the 10 concepts, and 50 images are generated per prompt. Background into Diffusion Models Latent diffusion models (LDMs) ( 29 ) rely on conditional inputs, such as text prompts ( 26 ) or images ( 25 ) , to guide the generation of images, utilizing an encoder \Phi(\cdot) and a decoder \Psi(\cdot) . Custom diffusion models (CDMs) ( 11 ) extend LDMs by incorporating low-rank adaptation (LoRA) ( 36 ) to fine-tune pretrained diffusion models ( 29 ) for personalized concept learning. Given a personalized image-text pair (x,p) , the encoder \Phi(\cdot) maps x to a latent representation z , with z_{t} denoting the noisy latent feature at timestep t ( t=1,\dots,T ). The text encoder \Gamma(\cdot) maps the text prompt p to a textual embedding c=\Gamma(p) . The objective for learning a personalized concept {(x,p)} at timestep t is defined as: \mathcal{L}_{CDM}=\mathbb{E}_{z\sim\Phi(x),c,\varepsilon\sim\mathcal{N}(0,I),t}\left[\|\varepsilon-\varepsilon_{\theta^{\prime}}(z_{t}|c,t)\|_{2}^{2}\right] (4) where \varepsilon_{\theta^{\prime}}(\cdot) represents the denoising UNet ( 29 ; 26 ) that gradually denoises z_{t} by estimating the Gaussian noise \varepsilon\sim\mathcal{N}(0,I) . The parameter set \theta^{\prime} corresponds to \theta^{\prime}=\theta_{0}+\Delta\theta , where \theta_{0}=\{W_{l}^{0}\}_{l=1}^{L} denotes the pretrained weights in LDMs, and \Delta\theta=\{\Delta W_{l}\}_{l=1}^{L} corresponds to the LoRA-updated parameters ( 13 ; 17 ) . Here, W_{l}^{0},\Delta W_{l}\in\mathbb{R}^{a\times b} are the pretrained and low-rank weight matrices in the l -th transformer layer of \theta^{\prime} , respectively, where a and b are matrix dimensions. Following ( 30 ) , the low-rank update \Delta W_{l} can be factorized as \Delta W_{l}=A_{l}B_{l} , where A_{l}\in\mathbb{R}^{a\times r} and B_{l}\in\mathbb{R}^{r\times b} with rank r\ll\min(a,b) . Note that the term “concept” refers to introducing the model to a new visual category, such as teaching it what a “zebra” looks like, as illustrated in Fig. 1 . Further Background into Hyperbolic Geometry Manifolds and Smooth Structures. A manifold is a topological space that is locally homeomorphic to Euclidean space. More precisely, an n -dimensional manifold M is a space where each point x\in M has a neighborhood that can be mapped bijectively and continuously (with a continuous inverse) to an open subset of \mathbb{R}^{n} . This structure allows for the analysis of complex, curved spaces using local coordinates. When the transition maps between overlapping coordinate charts are differentiable, the manifold is called a smooth manifold . Smooth manifolds provide the foundational framework for modern differential geometry, enabling calculus to be performed on non-Euclidean spaces. Riemannian Manifolds. A Riemannian manifold (M,g) is a smooth manifold M equipped with a smoothly varying inner product g_{x} on the tangent space T_{x}M at each point x\in M . The Riemannian metric g enables the measurement of geometric quantities such as the lengths of curves, angles between vectors, and notions of local curvature. The metric induces a distance function on the manifold, turning it into a metric space. Euclidean space \mathbb{R}^{n} is a simple example where the metric is constant and flat. In contrast, on general Riemannian manifolds, the curvature may vary spatially and fundamentally influences the geometry and topology of the space. Hyperbolic Geometry. Riemannian manifolds with constant negative curvature are called hyperbolic manifolds . These spaces exhibit properties that sharply contrast with those of flat (Euclidean) or positively curved (spherical) manifolds, such as exponential volume growth and distinct geodesic behavior. The canonical example is hyperbolic space \mathbb{H}^{n} , which can be realized through various models. The Poincaré ball model maps \mathbb{H}^{n} to the open unit ball in \mathbb{R}^{n} , preserving angles while distorting distances. Alternatively, the Lorentz model (also called the hyperboloid model) embeds \mathbb{H}^{n} as a submanifold of \mathbb{R}^{n+1} equipped with a Lorentzian inner product. This model is particularly useful in machine learning due to its closed-form geodesic computations and efficient optimization properties. Lorentzian inner product: Let \langle\cdot,\cdot\rangle denote the standard Euclidean inner product in \mathbb{R}^{n} . The Lorentzian inner product for vectors x,y\in\mathbb{R}^{n+1} , is defined as: \langle x,y\rangle_{\mathcal{L}}=\langle\mathbf{x}_{\text{space}},\mathbf{y}_{\text{space}}\rangle-x_{\text{time}}y_{\text{time}} . This inner product induces the Lorentzian norm \|x\|_{\mathcal{L}}=\sqrt{|\langle x,x\rangle_{\mathcal{L}}|} . The Lorentz model of hyperbolic space with constant negative curvature k is then given as : \mathcal{L}^{n}_{k}=\left\{x\in\mathbb{R}^{n+1}:\langle x,x\rangle_{\mathcal{L}}=-\frac{1}{k},\ x_{\text{time}}>0\right\}) . Each point x\in\mathcal{L}^{n}_{k} satisfies the constraint, x_{\text{time}}=\sqrt{\frac{1}{k}+\|\mathbf{x}_{\text{space}}\|^{2}} . Geodesics. The shortest path between two points on a Riemannian manifold is defined as a geodesic . In the Lorentz model of hyperbolic space, geodesics are defined as the intersections between the hyperboloid and hyperplanes that pass through the origin of the ambient space \mathbb{R}^{n+1} . The Lorentzian distance between two points x,y\in\mathcal{L}^{n} is given by, d_{\mathcal{L}}(x,y)=\frac{1}{\sqrt{k}}\cosh^{-1}(-k\langle x,y\rangle_{\mathcal{L}}) (5) Exponential and Logarithmic Maps. The exponential map at point z maps tangent vectors v\in T_{z}\mathcal{L}^{n} onto the manifold \mathcal{L}^{n} , describing how the tangent space folds back onto the manifold. Its inverse, the logarithmic map , projects a point x\in\mathcal{L}^{n} back to the tangent space. \texttt{expm}_{z}(v)=\cosh(\sqrt{k} \|v\|_{\mathcal{L}})z+\frac{\sinh(\sqrt{k} \|v\|_{\mathcal{L}})}{\sqrt{k}\|v\|_{\mathcal{L}}}v (6) \texttt{logm}_{z}(x)=\frac{\cosh^{-1}(-k\langle z,x\rangle_{\mathcal{L}})}{\sqrt{(k\langle z,x\rangle_{\mathcal{L}})^{2}-1}}\text{proj}_{z}(x) (7) In this paper, we use the origin O=[\mathbf{0},\sqrt{1/k}] of the hyperboloid as the reference point for both maps. Tangent Space. At any point z\in\mathcal{L}^{n} , the tangent space T_{z}\mathcal{L}^{n} forms a Euclidean subspace of \mathbb{R}^{n+1} and contains all vectors orthogonal to z under the Lorentzian inner product: T_{z}\mathcal{L}^{n}=\{v\in\mathbb{R}^{n+1}\mid\langle z,v\rangle_{\mathcal{L}}=0\} (8) Datasets We evaluate our approach on three distinct datasets, each curated to ensure high semantic clarity and alignment with our objectives. For the CIFC dataset, we directly adopt the benchmark provided in the reference paper, which includes conceptually rich visual cards that serve as a challenging ground for hallucination detection. For the ImageNet subset, we manually select 3–5 images per class across 20 classes, ensuring that the primary object of interest is prominently visible, well-centered, and unobstructed in each image. This careful selection mitigates background noise and ambiguous visual cues that could mislead the model during training or evaluation. Finally, for the CelebA dataset, we extract a subset comprising approximately 10 distinct identities, choosing 3–5 representative images per identity. Selection criteria included clear frontal facial orientation, consistent lighting, and minimal occlusion to maintain identity coherence across samples. This hand-picked curation across datasets ensures high-quality supervision signals, especially important in training models to reduce semantic hallucinations and enforce visual-textual alignment. Figure 11: Datasets. Class-wise representative images from the datasets used in our experiments. More Qualitative Results Figures 12 , 13 , and 14 present additional qualitative comparisons of generations from FLLP (Ours) against prior methods on CIFC ( 7 ) , CelebA ( 22 ) , and ImageNet ( 5 ) , respectively. Across all datasets, we observe consistent failure modes in TI ( 8 ) and CIDM ( 7 ) , notably the loss of subject identity and the emergence of extraneous or semantically irrelevant artifacts—often appearing in multiplicity. In contrast, FLLP (Ours) demonstrates stronger identity preservation and significantly reduced artifact generation, yielding image outputs with more desirable traits. Figure 12: Qualitative results on the CIFC dataset. We compare the generated images in TI ( 8 ) , CIDM ( 7 ) and FLLP (Ours). The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image. The hyperbolic tree indicates the parent chain of concepts that the model traversed in learning the incoming concept. Figure 13: Qualitative results on the CelebA dataset. We compare the generated images in TI ( 8 ) , CIDM ( 7 ) and FLLP (Ours). The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image. The hyperbolic tree indicates the parent chain of concepts that the model traversed in learning the incoming concept. Figure 14: Qualitative results on the ImageNet dataset. We compare the generated images in TI ( 8 ) , CIDM ( 7 ) and FLLP (Ours). The red and green arrows indicate regions of undesirable and desirable qualities, and their reasons are stated below each image. The hyperbolic tree indicates the parent chain of concepts that the model traversed in learning the incoming concept. Limitations Although CLIP ( 28 ) is a relatively older evaluation metric, it remains prevalent due to its computational efficiency, annotation-free nature, and scalability to large datasets. We recognize its limitations—particularly its reliance on global image-text similarity, which may fail to capture fine-grained or personalized attributes (e.g., “a dirty car”, “a yellow tree”). As shown in Figures 12 , 13 , and 14 , our prompts are intentionally coarse to align with our primary objective of evaluating catastrophic forgetting and inter-concept interactions, rather than fine-grained image fidelity. However, we acknowledge the need for a better metric. Furthermore, while FLLP (Ours) selectively utilizes optimal parent groups rather than all of the previously learned concepts, the overall parameter count remains unchanged when compared to CIDM ( 7 ) . Although this does not hinder performance, we note that no parameter reduction is achieved through this selective mechanism. Finally, our approach requires tuning of the threshold parameter \beta to maximize performance. However, this is the sole hyperparameter involved.