DuoLoRA : Cycle-consistent and Rank-disentangled Content-Style Personalization
Aniket Roy, Shubhankar Borse, Shreya Kadambi, Debasmit Das, Shweta Mahajan, Risheek Garrepalli, Hyojin Park, Ankita Nayak, Rama Chellappa, Munawar Hayat, Fatih Porikli · 2025
Read DuoLoRA : Cycle-consistent and Rank-disentangled Content-Style Personalization by Aniket Roy, Shubhankar Borse, Shreya Kadambi, Debasmit Das, Shweta Mahajan, Risheek Garrepalli, Hyojin Park, Ankita Nayak, Rama Chellappa, Munawar Hayat, Fatih Porikli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13206.txt.
Introduction
Aniket Roy and co-authors present DuoLoRA, a method to personalize image models for content and style at once, in a research paper from 2025. Users may want a subject from a few photos rendered in a style from a few others. Training split adapters for content and style, then merging them, is a promising route, but past merges treat the two as independent.
Content and style in fact intertwine. DuoLoRA merges inside the rank dimension of the adapters, cutting learnable parameters. Layer priors from the base model guide how much rank each layer gives to content or style. A cycle-consistency loss refines the merge. Tests show cleaner joint personalization from few examples.
Catalog record
| CREATOR | Aniket Roy, Shubhankar Borse, Shreya Kadambi, Debasmit Das, Shweta Mahajan, Risheek Garrepalli, Hyojin Park, Ankita Nayak, Rama Chellappa, Munawar Hayat, Fatih Porikli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13206
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DuoLoRA : Cycle-consistent and Rank-disentangled Content-Style Personalization Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Aniket Roy, Shubhankar Borse, Shreya Kadambi, Debasmit Das, Shweta Mahajan, Risheek Garrepalli, Hyojin Park, Ankita Nayak, Rama Chellappa, Munawar Hayat, Fatih Porikli Abstract We tackle the challenge of jointly personalizing content and style from a few examples. A promising approach is to train separate Low-Rank Adapters (LoRA) and merge them effectively, preserving both content and style. Existing methods, such as ZipLoRA, treat content and style as independent entities, merging them by learning masks in LoRA’s output dimensions. However, content and style are intertwined, not independent. To address this, we propose DuoLoRA—a content-style personalization framework featuring three key components: (1) rank-dimension mask learning, (2) effective merging via layer priors, and (3) Constyle loss, which leverages cycle-consistency in the merging process. First, we introduce ZipRank, which performs content-style merging within the rank dimension, offering adaptive rank flexibility and significantly reducing the number of learnable parameters. Additionally, we incorporate SDXL layer priors to apply implicit rank constraints informed by each layer’s content-style bias and adaptive merger initialization, enhancing the integration of content and style. To further refine the merging process, we introduce Constyle loss, which leverages the cycle-consistency between content and style. Our experimental results demonstrate that DuoLoRA outperforms state-of-the-art content-style merging methods across multiple benchmarks. 1 Introduction Figure 1 : Content and style personalization using DuoLoRA provides (1) adaptive rank flexibility, (2) significantly lesser trainable parameters, and (3) better content-style merging. Personalizing text-to-image diffusion models [ 27 , 31 , 6 ] has recently attracted significant interest due to their broad range of applications. These models can produce images with desired content and style, overcoming the limitations of earlier approaches that required extensive training data for neural style transfer [ 13 ] . They can generate specific images from just one or two reference images, but blending both content and style remains challenging. Recently, parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adapters (LoRA) [ 17 ] have become popular. These methods can capture unique characteristics with a small amount of data. Consequently, content-style personalization can be reframed as a LoRA merging task, where individually trained LoRAs are combined to achieve efficient content-style blending. For example, ZipLoRA [ 29 ] merges content and style by learning masks over the LoRA output dimensions, assuming that content and style are independent concepts. This approach provides adequate control over the diffusion model. However, the requirement for fine-tuning during inference is a drawback. Additionally, using the same rank for both content and style LoRAs may not be optimal, as content and style may have different representational requirements depending on each layer. In this work, we ask: How can we enable adaptive rank flexibility to reduce fine-tuning costs while increasing the separation between content and style distributions? To achieve adaptive rank flexibility across the layers of the diffusion UNet, we analyze the SDXL model architecture [ 3 , 10 ] . We observe that UNet layers, with smaller resolutions, primarily influence content generation, suggesting that content merges should have higher rank in these layers than style merges. Conversely, the layers, with larger resolutions, are more crucial for style generation, and the rank constraint should be adjusted accordingly [ 3 , 10 ] . Within this context, we introduce DuoLoRA, a framework for effective content-style merging, composed of three components: (1) ZipRank, which learns masks within the rank dimension, (2) improved merging via layer priors, and (3) Constyle loss, which leverages cycle-consistency across content and style. In ZipRank, we propose learning masks in the rank dimension rather than the output dimension, allowing for adaptive rank adjustment and a substantial reduction in learnable parameters. To facilitate layer-wise rank-adaptive merging, we apply rank constraints during content-style merging, informed by our observations as prior knowledge. This rank constraint is formulated as a nuclear norm minimization problem under a l1 sparsity constraint, improving content-style blending with fewer parameters. To further disentangle content and style, we introduce Constyle loss, which leverages cycle-consistency across content and style. This approach, inspired by CycleGAN’s [ 45 ] treatment of content and style as separate domains optimized through domain translation with minimized reconstruction loss, enables balanced content-style merging. Unlike ZipLoRA, which treats content and style independently, our cycle-consistency loss accounts for their interdependent nature, resulting in improved content-style blending. In summary, the contributions of this paper are as follows: • We address content-style personalization as a LoRA merging problem, where we propose learning masks in the rank dimension instead of the output dimension, allowing for adaptive rank flexibility with significantly fewer learnable parameters. • We analyze the SDXL architecture’s layer prior information, finding that layers with lower resolutions primarily contribute to content generation, while layers with higher resolutions focus on style generation. Based on these insights, we introduce explicit rank constraints through nuclear norm minimization under a sparsity constraint to improve merging. • We introduce Constyle loss, leveraging the cycle-consistency across content and style, and we validate this approach across various benchmarks. 2 Related work Personalization. Personalizing text-to-image diffusion models has recently attracted significant attention. Early approaches, such as Textual Inversion [ 11 ] , focus on learning a text token that represents a particular concept, while Dreambooth [ 27 ] updates network parameters for personalization. Custom Diffusion [ 22 ] makes the process more efficient by fine-tuning only the cross-attention modules. However, these methods typically handle only a single concept or object. In contrast, some approaches aim to personalize both content and style. StyleDrop [ 31 ] uses the Muse transformer network to align the style of generated images with a reference image. Other recent style learning methods are - Rb-modulation [ 26 ] , Instantstyle [ 36 ] , IP-adapter [ 42 ] , Magic-insert [ 28 ] , StyleAlign [ 16 ] , LoRA-composer [ 41 ] , Paircustomization [ 20 ] , [ 30 , 40 , 39 , 43 , 25 , 17 , 7 , 4 , 2 , 37 , 8 , 34 , 36 , 38 , 19 , 24 , 21 , 1 , 14 , 18 , 33 , 9 , 35 , 5 , 32 , 23 ] . LoRA merging. LoRA [ 17 ] has proven particularly effective for learning from small datasets. Individual LoRA models are trained for each concept or style, and these concepts can then be combined through LoRA merging. Methods like Concept-Sliders [ 12 ] and ControlNet [ 44 ] also utilize LoRA merging, though their approaches are primarily suited for text-based editing. Recently, Shah et al. [ 29 ] introduced a method for LoRA merging by learning orthogonal masks in the output space of the layer. Layer priors. We analyze the SDXL architecture and uncover some intriguing insights. Specifically, we observe that the earlier layers in the diffusion UNet contribute more to local edits, such as content generation, while the later layers influence more global edits. Other baseline approaches include naive parameter merging [ 29 ] and B-LoRA [ 10 ] . Recent studies have also examined how information is encoded within the SDXL architecture, demonstrating its utility for fast image editing [ 3 ] . These findings inspire us to investigate layer prior