LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs
Jiaze Wang Rui Chen Haowang Cui · 2025
Read LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs by Jiaze Wang Rui Chen Haowang Cui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19939.txt.
Introduction
Text-to-image models often drop an object or misplace it when a prompt gets crowded. Jiaze Wang and co-authors put a multimodal language model in charge of the layout. Their framework, LLMControl, uses the model as a global controller. It arranges where things go, enriches thin descriptions, and binds each attribute to the right object.
Those control signals flow into the image generator's denoising process. New sampling rules refocus attention so regions match the plan. Tests across several base generators show quality on par with the best specialized methods. The hard cases, many objects in tricky arrangements, are where the controller earns its keep. The work treats composition as a planning problem first and a painting problem second.
Catalog record
| CREATOR | Jiaze Wang Rui Chen Haowang Cui |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19939
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs Jiaze Wang Rui Chen Haowang Cui Abstract Recent spatial control methods for text-to-image (T2I) diffusion models have shown compelling results. However, these methods still fail to precisely follow the control conditions and generate the corresponding images, especially when encountering the textual prompts that contain multiple objects or have complex spatial compositions. In this work, we present a LLM-guided framework called LLM_Control to address the challenges of the controllable T2I generation task. By improving grounding capabilities, LLM_Control is introduced to accurately modulate the pre-trained diffusion models, where visual conditions and textual prompts influence the structures and appearance generation in a complementary way. We utilize the multimodal LLM as a global controller to arrange spatial layouts, augment semantic descriptions and bind object attributes. The obtained control signals are injected into the denoising network to refocus and enhance attention maps according to novel sampling constraints. Extensive qualitative and quantitative experiments have demonstrated that LLM_Control achieves competitive synthesis quality compared to other state-of-the-art methods across various pre-trained T2I models. It is noteworthy that LLM_Control allows the challenging input conditions on which most of the existing methods fail. 1 Introduction Figure 1 : The figure shows the results generated by our method. The text are the complex prompts used in the generation. the left column are the spatial image conditions, the middle column are the polygon layouts generated by our method, and the right column are the generated images. 2 Introduction With the emergence of a range of diffusion models [ 13 , 44 , 43 ] represented by Stable Diffusion [ 34 , 37 , 12 ] , T2I task has taken on a whole new look. Using T2I diffusion models, we can now generate high-quality images that meet mental expectations with simple inputs. Diffusion models generate results by iteratively transforming an initial simple distribution into a complex high quality distribution. However, there are still some problems in the T2I diffusion generation task. For users, it is difficult to accurately describe the desired effect by text alone and text that is too complex and lengthy is sometimes difficult to be directly understood by diffusion models. On top of the underlying model, more and more works [ 29 , 31 , 54 , 51 , 32 , 8 , 45 , 2 ] are introducing additional spatial image to achieve more precise control. But we find that the trained model tends to prioritize image-space conditions over textual descriptions, probably because the input-output image pairs are tightly spatially aligned, which often leads to the results that are not aligned with the user’s textual instructions. Moreover, while methods such as ControlNet [ 29 ] and T2I Adapter [ 8 ] allow users to provide more precise spatial control of pre-trained T2I diffusion models by providing various types of image conditions as well as textual descriptions, these methods [ 40 , 14 , 7 , 8 ] often require the training of different add-on modules for different types of spatial images, leading to unnecessary repetitive training and expensive computational costs. Figure 2 : The figure shows the architecture of our approach. Our method consists of three stages. Firstly, we utilize MLLM planning to generate tasks. Then, we encode to get the embedding and generate cross-attention masks. Finally, we utilize structure guidance and appearance guidance to generate the final image. To address the above limitations, we propose LLMControl, a controllable T2I diffusion method. Previous work [ 52 , 22 , 50 , 1 ] has shown that diffusion models usually predict low-frequency content at the beginning of the denoising process and high-frequency details at the final stage of the process. Low-frequency semantics correspond to main objects and locations, while high-frequency semantics are related to detailed attributes. So we will use multimodal LLM (MLLM) [ 24 , 6 , 25 , 26 , 10 ] as a global task planner to guide the generation process through three levels of layout, structure and appearance with user input of textual prompts and spatial image conditions. Some of our generated image results are shown in Figure 1 . Specifically, LLMControl has three stages, As shown in figure 2 . In the Planning Stage, MLLM can decompose the generation task into a combination of subtasks by splitting the complex text input into multiple objects, each with its own appearance description and layout. The results of subtasks will eventually be combined into the final image. This is followed by an Encoding Stage, where the layout and appearance description of each object will be encoded as embeddings and connected together, and then fed into the diffusion model’s cross-attention. In contrast to previous work [ 49 , 48 , 54 , 57 , 55 , 9 ] that used rectangles as layouts, we propose a new visual representation that describes the position of objects with polygons. Finally, there is a Feature Stage, where we obtain diffusion features of the spatial image by DDIM Inversion [ 19 ] , which will be used for structure guidance. Notably, we use pre-trained MLLMs and diffusion models to reduce training costs. Our approach is also compatible with different types of spatial images without the need to train specific components for different additional image inputs. LLMControl performs well under challenging input conditions. It obtains competitive image synthesis quality while providing stronger image-text alignment and supporting a broader set of control signals. In summary, our contributions include: 1) In this paper, we proposed a new controllable T2I generation method, LLMControl, which guides the generation process by hierarchically utilizing layout, structure and appearance to achieve precise alignment and high level of image quality. 2) We proposed a new visual primitive representation, i.e., each object in an image is represented by an irregular polygonal layout with textual descriptions. This preserves richer and more precise visual information, and also facilitates human and MLLM understanding and construction. 3) We extracted structural features of the spatial image condition and injected them during the generation process, thus guiding the generated image to follow the structure of the spatial image. The spatial image conditions of various modalities have a unified representation under our design without having to train multiple components 4) We have conducted extensive experiments on various benchmarks, showing that LLMControl is comparable to previous SOTA methods. 3 Related work 3.1 T2I generation models Compared to GAN [ 38 , 46 , 15 ] and autoregressive models [ 17 , 11 , 18 ] , diffusion models [ 35 , 13 , 44 , 43 , 19 ] have become more popular recently due to the high quality and diversity of the images they generate. So far, many image generation methods based on diffusion models have been proposed. Among them, Stable Diffusion [ 34 , 37 , 12 ] is popular due to its strong performance and robust open-source status. It usually employs the latent diffusion model (LDM) , which allows the denoising process to operate in a low-dimensional latent space, thus reducing the computational cost. Besides, a number of fine-tuning techniques [ 31 , 32 , 40 , 36 , 33 , 56 , 1 , 7 , 58 , 53 ] have been proposed that enhance the