SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

Javad ParsaUppsala University, ETH Zurich, Enis SimsarETH ZurichAmir JoudakiETH ZurichThomas HofmannETH ZurichAndré M. H. TeixeiraUppsala University, Sweden · 2026

Read SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation by Javad ParsaUppsala University, ETH Zurich, Enis SimsarETH ZurichAmir JoudakiETH ZurichThomas HofmannETH ZurichAndré M. H. TeixeiraUppsala University, Sweden (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22743.txt.

Introduction

Parsa and four colleagues offer SeqLoRA, a method for teaching image generators many personal concepts, such as objects, people, or styles, without them bleeding into each other. It trains each new low-rank adapter orthogonally to earlier ones, using a two-level optimization that keeps both adapter factors free. Tests with up to 101 concepts on Stable Diffusion improved identity preservation and cut cross-concept leakage, with no separate fusion step.

Catalog record

CREATORJavad ParsaUppsala University, ETH Zurich, Enis SimsarETH ZurichAmir JoudakiETH ZurichThomas HofmannETH ZurichAndré M. H. TeixeiraUppsala University, Sweden
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122317
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22743

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation Javad ParsaUppsala University, ETH Zurich, Enis SimsarETH ZurichAmir JoudakiETH ZurichThomas HofmannETH ZurichAndré M. H. TeixeiraUppsala University, Sweden Abstract Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models, but composing multiple custom concepts remains challenging due to representation interference. Existing modular methods either rely on expensive post-hoc fusion or freeze adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization. Theoretically, we establish strong convergence guarantees for our algorithm and model the residual layer activations as a matrix sub-Gaussian process to derive high-probability bounds on catastrophic forgetting. We further prove that learning the LoRA basis from data minimizes residual interference energy more effectively than frozen-basis methods. Experiments on multi-concept image generation demonstrate that SeqLoRA improves identity preservation and scalability across up to 101 concepts, while avoiding costly fusion and reducing attribute interference in composed generations. 1 Introduction Text-to-image diffusion models have transformed visual content generation by allowing users to create high-quality, diverse images directly from natural-language prompts ( 13 ; 27 ; 29 ; 25 ) . These models are trained to invert a progressive noising process, gradually converting random noise into a coherent image that aligns with a textual description. Thanks to their stable optimization behavior and strong ability to capture diverse data modes, diffusion models now support a wide range of applications, including artistic creation, media production, data augmentation, and scientific visualization ( 23 ; 19 ) . Large-scale text-to-image systems such as Stable Diffusion ( 27 ) and DALL·E ( 25 ) have seen broad adoption, while parameter-efficient fine-tuning methods, especially Low-Rank Adaptation (LoRA) ( 16 ) , have made it feasible for individual users to adapt these foundation models to new visual concepts using only a small set of reference images ( 28 ; 10 ) . While single-concept customization has become highly mature ( 28 ; 10 ; 16 ) , a key challenge arises when users want to generate images that include multiple personalized concepts at once, for instance, two specific pets interacting in the same scene, or a particular artistic style combined with a custom object ( 17 ; 11 ) . The central scientific issue is representation interference : because concepts are usually learned separately, their corresponding adaptations may occupy overlapping regions in the model’s parameter space ( 22 ; 18 ) . When these adaptations are combined at inference time, the shared directions can lead to attribute entanglement , where identity-related features from one concept bleed into another, resulting in mixed attributes, weakened identities, or even incoherent generations ( 22 ; 11 ) . This problem is especially pronounced for semantically similar concepts, such as different human faces, whose learned weight residuals tend to be highly correlated and lie in nearly the same subspaces ( 22 ) . As the number of concepts increases, the combinatorial growth in possible concept subsets makes it infeasible to train a separate model for every combination, motivating modular methods in which independently fine-tuned components can be composed at test time without retraining ( 11 ; 22 ) . Figure 1 : Qualitative comparison of multi-concept image generation across different methods for 32 concepts. We show the original concept images (Input) and 4 generated samples (seeds) for all methods. Several recent works have proposed strategies to mitigate concept interference, each offering partial solutions with distinct trade-offs. Joint training methods such as Custom Diffusion ( 17 ) and Break-A-Scene ( 2 ) optimize all concepts simultaneously within a shared model, which can reduce interference but requires concurrent access to all concept data, a requirement that is unscalable to large concept libraries and raises data privacy concerns in multi-user settings. Gradient fusion approaches, exemplified by Mix-of-Show ( 11 ) , allow independent fine-tuning of each concept and then merge the resulting models through an optimization-based fusion step to preserve identity; however, the fusion must be re-executed for every new combination of concepts, preventing truly modular composition where individual adapters can be added or removed on the fly. Orthogonal Adaptation ( 22 ) takes a different approach by sampling each concept’s LoRA basis {\bf B}_{i} from a shared orthogonal matrix and freezing it during training, optimizing only the coefficient matrix {\bf A}_{i} . This guarantees approximate orthogonality between concepts and enables instant merging via simple summation, but freezing {\bf B}_{i} restricts the feasible space of the adaptation and limits per-concept fidelity. In the continual learning literature, InfLoRA ( 18 ) designs task-specific LoRA subspaces for image classification that are orthogonal to the gradient subspaces of previously learned tasks, achieving strong stability–plasticity trade-offs for sequential classification; however, it similarly freezes the basis matrix for each task, inheriting the same expressiveness limitation. A common thread across these methods is the tension between orthogonality, which is required to prevent interference, and expressiveness, which is required for high-fidelity adaptation: existing methods do not simultaneously permit both LoRA factors to be optimized freely while enforcing the subspace orthogonality necessary for low interference composition. In this work, we resolve this tension by formulating multi-concept generation as a constrained continual learning problem. As a preview of our results, Figure 1 visualizes a qualitative comparison on a 32-concept task. Notably, for challenging concepts like ”wearableglasses”, baseline methods suffer from severe attribute entanglement, whereas SeqLoRA (our method) successfully preserves distinct concept identities. Contributions. This paper makes the following contributions: • We introduce Sequential regularized LoRA (SeqLoRA), which jointly optimizes both LoRA factors via bilevel optimization while enforcing subspace orthogonality through a closed-form projection, resolving the expressiveness–interference trade-off. • We provide theoretical analyses establishing: (a) monotone descent and convergence to a critical point of the constrained problem (Theorem 1 ), and (b) derive high-probability catastrophic forgetting bound showing that learning the LoRA bases reduces residual interference compared to frozen bases (Theorem 2 ). • We evaluate SeqLoRA on multi-concept generation with up to 101 concepts, demonstrating state-of-the-art identity preservation and scalability beyond the capacity of existing fusion-based methods. Structure of the paper. The remainder of the paper is organized as follows. Section 2 introduces the continual-learning model, the LoRA parameterization, and the interference problem. Section 3 develops the bilevel optimization formulation, derives the closed-form projection, and presents the SeqLoRA algorithm with its theoretical analysis. Section 4 reports experimental results. Section 5 discusses limitations and broader impact. 2 Problem Formulation 2.1 Continual Learning Model We consider a sequential concept stream \tau_{1},\dots,\tau_{T} , where each task \tau_{i} provides a small reference dataset \mathcal{D}_{i}=\{(\mathbf{x}_{i,m},c_{i})\}_{m=1}^{M_{i}} pairing M_{i} reference images with a concept-specific text token c_{i} . Only \mathcal{D}_{i} is accessible when task \tau_{i} arrives; prior datasets are discarded. The goal is a compact, composable representation per concept such that any subset of the T concepts can be jointly rendered at inference time without retraining, balancing plasticity (adapting to new concepts) and stability (preserving previous ones). 2.2 Single-Concept Fine-Tuning with LoRA LoRA ( 16 ) has emerged as the dominant strategy for parameter-efficient fine-tuning of diffusion models. Rather than updating the full weight matrix {\bf W}_{i}\in{\mathbb{R}}^{n\times m} of every attention layer, LoRA introduces a low-rank residual for each concept i : {\bf W}_{i}={\bf W}_{0}+{\bf A}_{i}{\bf

Read the complete text