Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation
Shuhong Zheng Aashish Kumar Misraa Yu-Teng Li, Yu-Jhe Li Igor Gilitschenski · 2026
Read Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation by Shuhong Zheng Aashish Kumar Misraa Yu-Teng Li, Yu-Jhe Li Igor Gilitschenski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26111.txt.
Introduction
Zheng and four colleagues improve image generation that must keep a subject's look while obeying text prompts. Encoding text and photos apart limits joint reasoning and causes copy-paste flaws. Systems that link joint models to diffusion models follow prompts better but neglect identity. The authors feed diffusion models with joint models that read text and reference photos together, plus VAE-based identity cues. A dual layer aggregation module pools multi-level traits for guidance, and staged denoising balances prompt meaning against fine identity detail.
Catalog record
| CREATOR | Shuhong Zheng Aashish Kumar Misraa Yu-Teng Li, Yu-Jhe Li Igor Gilitschenski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69575 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation Shuhong Zheng Aashish Kumar Misraa Yu-Teng Li, Yu-Jhe Li Igor Gilitschenski Abstract Subject-driven image generation aims to synthesize new images that preserve the identity of the given subject while following textual instructions. Existing approaches often encode text and reference images separately. This limits cross-modal reasoning abilities and causes copy-paste artifacts. Recent frameworks that connect multimodal models and diffusion models improve instruction following, but largely overlook identity preservation. To address these limitations, we condition diffusion models on Multimodal Large Language Models (MLLMs) that jointly encode text and reference images, and augment it with VAE - based identity conditioning. A novel Dual Layer Aggregation (DLA) module is designed to aggregate multi - level MLLM features for optimal conditioning, and a multi - stage denoising strategy is applied to progressively balance the semantic information from MLLM and fine - detail identity from VAE during inference. Extensive experiments demonstrate that our approach harmonizes multimodal understanding with identity preservation, mitigates copy-paste issues, and achieves superior performance regarding human preference on subject-driven image generation. Our project website is available at https://zsh2000.github.io/squeeze-mllm-subject-gen/ . $\dagger$ $\dagger$ footnotetext: Joint Advising * * footnotetext: Work done in Adobe 1 Introduction Subject-driven image generation aims to synthesize new content while preserving the visual identity of a specific subject. Early approaches [ 98 , 64 , 19 , 48 , 106 , 2 , 9 , 65 ] , such as DreamBooth [ 83 ] and Textual Inversion [ 23 ] , personalize pretrained diffusion models via per-subject fine-tuning, achieving strong identity fidelity at the cost of scalability. Subsequent works [ 17 , 87 , 38 , 67 , 66 , 127 , 76 ] adopt reference-image conditioning to avoid retraining, where models like IP-Adapter [ 123 ] extract subject features at inference time. More recent efforts [ 110 , 6 , 72 , 22 , 124 , 18 ] further enhance zero-shot subject generalization through VAE-based (Variational Autoencoder-based [ 45 ] ) token conditioning. However, these pipelines still process text and reference images separately, limiting multimodal understanding and often producing copy-paste artifacts or identity drift on complex prompts. In parallel, multimodal large language models (MLLMs) [ 61 , 62 ] have demonstrated strong abilities in joint text-image reasoning and structured control [ 90 ] . Systems [ 15 , 89 ] such as DreamEngine [ 10 ] , Qwen-Image [ 107 ] , and EasyRef [ 131 ] integrate MLLMs into diffusion decoders to parse interleaved multimodal instructions, enabling more flexible prompt interpretation. Yet, these designs typically rely only on the MLLM’s final-layer features ( e.g., Qwen-Image, EasyRef), or combine ViT features which contain fine details, with final-layer outputs via scalar mixing ( e.g., DreamEngine). These models often neglect fine-grained visual cues which are crucial for identity, thereby leading to suboptimal identity preservation. In this work, we unify these two directions by introducing an MLLM-driven subject conditioning framework that jointly encodes text and reference images within a shared multimodal space, and enhances ID preservation with VAE conditioning. This joint encoding enables the model to perform multimodal reasoning and coherently preserve subject identity, beyond the representational limits of pure VAE-based encoders. However, this unification is non-trivial due to the different feature structures of text and image tokens in MLLMs. The discrepancy between text and image features makes it fundamentally inadequate to directly fuse modalities or rely on a single-layer representation for conditioning. To effectively align MLLM embeddings with diffusion features, we design an innovative Dual Layer Aggregation (DLA) mechanism, that adopts layerwise attention pooling to separately aggregate text and visual embeddings. Instead of conditioning solely on the MLLM’s final layer feature, the DLA takes the aggregated features from all transformer layers in the MLLM as input, to fully leverage its multimodal prompt understanding capability. We also justify the mechanism of aggregation by analyzing the roles and effectiveness of different layer groups ( i.e., early, middle, and late layers) within MLLM in the experimental study. Figure 1: Benefits of leveraging MLLMs for subject-driven generation. MLLMs mitigate the copy-paste issue within VAE-based methods, and enables the multimodal understanding of the subject-driven generation pipeline by jointly modeling input image and text, while VAE-based methods encode them separately. In addition, directly combining MLLM embeddings with VAE-based identity enhancement can cause embedding conflicts, as both contain overlapping visual representations. To reconcile these signals, a two-stage training strategy is invented to first enable multimodal conditioning from MLLM, before combining the optimization with the high-frequency identity details from VAE features. To further balance the multimodal conditioning from the MLLM and the identity details provided by the VAE, we propose a multi-stage denoising strategy: the diffusion model first denoises under MLLM guidance to establish global semantics, then jointly refines with both modalities, and finally focuses on VAE-conditioned fine details. As shown in Figure 1 , this staged process effectively harmonizes the two embedding sources, alleviating copy-paste artifacts common in VAE-based pipelines, while providing richer reasoning ability and instruction-aware, identity-preserving generation compared to existing frameworks. Our contributions can be summarized as follows: we propose a Dual Layer Aggregation (DLA) module to aggregate text and visual embeddings across MLLM layers for improved conditioning, along with a multi-stage denoising strategy that balances semantic reasoning and fine-grained identity during generation. Also, we provide a detailed analysis of MLLM layer representations and their roles in diffusion conditioning under different fusion strategies. Extensive experiments demonstrate competitive performance in multimodal understanding and identity preservation over prior subject-driven methods. 2 Related Work Subject-driven Generation focuses on preserving the identity or visual characteristics of a specific subject within the synthesized images. Early optimization-based approaches [ 12 , 32 , 24 , 1 ] such as DreamBooth [ 83 ] , Textual Inversion [ 23 ] , and LoRA [ 35 ] adapt pretrained diffusion models to new identities by introducing subject-specific parameters, but require costly per-subject fine-tuning. To eliminate this need, recent methods employ explicit reference encoders or adapters that extract identity features directly from input images and condition the diffusion process at inference time ( e.g. , IP-Adapter [ 123 ] , BLIP-Diffusion [ 50 ] ). Transformer-based diffusion decoders (DiT) have further incorporated such reference conditioning [ 58 , 36 ] through lightweight modules like IC-LoRA [ 37 ] . Subsequent research [ 39 , 51 , 116 , 59 , 44 ] enhances facial fidelity [ 77 , 117 , 112 , 52 , 95 , 104 ] , multi-reference composition [ 100 , 43 , 126 , 128 , 86 , 118 , 88 , 99 , 96 , 119 , 105 , 120 , 84 , 31 ] , computational efficiency [ 41 , 53 , 54 , 16 , 103 , 121 , 122 , 56 ] , and multimodal controllability [ 30 , 115 , 49 , 40 , 21 , 114 , 113 , 101 , 33 , 60 , 92 ] . Recently, UNO [ 110 ] , UMO [ 14 ] , USO [ 109 ] , and DreamO [ 69 ] achieve zero-shot generation conditioned by multiple images leveraging VAE-based token conditioning. However, these identity-preserving and control-oriented pipelines remain largely decoupled from large multimodal language models (MLLMs), lacking