Test-time Prompt Refinement for Text-to-Image Models
Mohammad Abdul Hafeez Khan, Yash Jain, Siddhartha Bhattacharyya, Vibhav Vineet · 2025
Read Test-time Prompt Refinement for Text-to-Image Models by Mohammad Abdul Hafeez Khan, Yash Jain, Siddhartha Bhattacharyya, Vibhav Vineet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22076.txt.
Introduction
Mohammad Abdul Hafeez Khan and co-authors refine the text prompt of an image making model at the time of use, in a loop that needs no new training of the model. Such models are touchy: a small change in the words can give a wrong or shaky image.
Each round, a large model that can see reads the fresh image next to the user's words. It spots what is off, like a missing object or a wrong trait, and writes a better prompt grounded in the real world. The loop runs again until the image fits. Tests across data sets show gains in fit and in how well the image hangs together.
Catalog record
| CREATOR | Mohammad Abdul Hafeez Khan, Yash Jain, Siddhartha Bhattacharyya, Vibhav Vineet |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-time Prompt Refinement for Text-to-Image Models Mohammad Abdul Hafeez Khan, Yash Jain, Siddhartha Bhattacharyya, Vibhav Vineet Abstract Text-to-image (T2I) generation models have made significant strides but still struggle with prompt sensitivity: even minor changes in prompt wording can yield inconsistent or inaccurate outputs. To address this challenge, we introduce a closed-loop, test-time prompt refinement framework that requires no additional training of the underlying T2I model, termed TIR . In our approach, each generation step is followed by a refinement step, where a pretrained multimodal large language model (MLLM) analyzes the output image and the user’s prompt. The MLLM detects misalignments (e.g., missing objects, incorrect attributes) and produces a refined and physically grounded prompt for the next round of image generation. By iteratively refining the prompt and verifying alignment between the prompt and the image, TIR corrects errors, mirroring the iterative refinement process of human artists. We demonstrate that this closed-loop strategy improves alignment and visual coherence across multiple benchmark datasets, all while maintaining plug-and-play integration with black-box T2I models. Figure 1 : Effectiveness of the proposed Test-time Image Refinement (TIR) approach in enhancing image generation quality. TIR iteratively refines input prompts based on the generated images to more accurately reflect the given text instructions. For each prompt, the left image shows the original DALL-E 3 output, while the right image presents the refined result produced by our approach (TIR), demonstrating improved content control and better adherence to the specified prompt. Best viewed in color. 1 Introduction Text-to-image (T2I) models have enabled users to create high-quality images from textual descriptions [ 40 , 38 , 37 , 39 , 22 ] . However, these models often fail when prompts require compositional reasoning or commonsense understanding. For instance, we observe that given the prompt “a horse riding an astronaut” , a T2I model often generates an astronaut riding a horse instead, as shown in Fig. 1 , thereby reversing the intended subject-object relationship. This occurs because the model defaults to more statistically frequent or visually plausible configurations learned during training, rather than faithfully interpreting the prompt’s semantic structure. Such errors reflect a broader inability to interpret uncommon spatial relations, especially when they conflict with learned priors or require understanding of syntactic cues, causal direction, or atypical physical dynamics. This reveals a core limitation of current T2I systems: they operate in an open-loop fashion and lack the ability to assess prompt plausibility or self-correct errors after generation. Even advanced models like DALL-E 3 [ 5 ] , Stable Diffusion [ 40 ] , and Flux [ 22 ] often produce images that are photorealistic yet semantically incorrect. This is especially evident in prompts involving spatial relationships, counting, negation, or temporal logic, where adherence to the prompt breaks down despite generating high-quality images. These challenges indicate that T2I models struggle with the implicit semantics of prompts unless such constraints are made explicit, which better aligns with training-time prompt-image supervision based on direct token-to-visual feature associations [ 36 , 38 , 40 ] . Existing solutions such as fine-tuning [ 23 , 10 ] , post-hoc editing [ 26 , 46 , 2 , 15 ] , and prompt engineering offer partial solutions but come with trade-offs. Training-based methods improve alignment but require costly, model-specific adaptation; post-hoc corrections often distort scene structure by editing images after generation; and prompt engineering shifts the burden onto users to craft highly-specific, exhaustive prompts. A fundamental challenge remains: Can we enable T2I models to self-correct misinterpretations iteratively, without additional training? As MLLMs like Qwen [ 44 , 49 ] and GPT-4o [ 16 ] continue to advance in multimodal reasoning, their understanding of text and images makes them well-suited to provide semantic feedback on the alignment between prompts and generated images. Leveraging this capability, we introduce Test-time Image Refinement (TIR), a closed-loop framework, illustrated in Fig. 2 . Instead of treating T2I generation as a one-shot process, TIR uses an MLLM in each iteration to assess misalignments between the user’s intent and the generated image, focusing on semantic reasoning failures such as spatial errors, attribute mismatches, incorrect object counts, etc. It then produces a refined prompt that explicitly corrects these issues while preserving contextual history from prior refinements. This history-aware feedback loop enables the T2I model to progressively converge toward the intended scene, without relying on handcrafted prompts, scene layouts, or bounding-box supervision. TIR requires no additional training or architectural modifications. Instead, it treats test-time compute as a resource and integrates seamlessly with off-the-shelf T2I models such as Stable Diffusion [ 40 ] , DALL-E 3 [ 5 ] , and Flux [ 22 ] , making it a flexible, plug-and-play solution for improving prompt adherence. Our key contributions are as follows: • MLLM-driven Test-time Refinement: We introduce an iterative refinement framework to improve prompt-image alignment without modifying the T2I model. • Training-Free, Plug-and-Play Adaptability: Our method requires no fine-tuning and is compatible with black-box APIs, making it applicable to any pretrained T2I model. • Empirical Validation: We validate our method across multiple datasets and T2I models, showing that TIR improves semantic fidelity, enables reasoning capabilities at test time and, in the future, it may help construct reasoning benchmarks for generative models. Overall, our work bridges the gap between the generative strength of T2I models and the need for adaptive, self-correcting systems. By leveraging MLLM-guided refinement at test time, we enable models to iteratively improve semantic alignment, mimicking the way human artists refine their work through feedback. Figure 2 : Test-time prompt refinement for Text-to-Image Models ( TIR ). Starting from a user-provided input prompt, we generate an image using a text-to-image model. A Multi-modal Large Language Model (MLLM) evaluates the consistency between the output image and the prompt, and refines the prompt based on feedback. We retain all previous prompts and feedback in memory to ground the image generation in the input instructions and iteratively improve the output. We repeat this cycle for a fixed number of iterations, progressively refining the prompt and generating an image that best aligns with the original user intent. 2 Literature Review 2.1 Text-to-Image (T2I) Models Text-to-image generation has evolved from GAN-based models [ 39 , 52 , 55 , 43 ] to transformer-based architectures [ 8 , 9 , 37 ] and diffusion-based models [ 38 , 33 , 40 , 41 ] , significantly improving generation quality and text fidelity. However, T2I models still struggle with compositional reasoning and prompt adherence, failing to capture fine-grained spatial relationships, attribute details, and multi-object interactions [ 3 , 31 ] . To improve controllability, researchers have explored structured conditioning signals such as keypoints, bounding boxes, scene graphs, and other domain-specific representations [ 6 , 25 , 32 , 19 , 50 , 18 , 51 , 17 , 53 ] . While effective, these methods rely on external annotations that may not always be available. Training-free alternatives leverage pretrained LLMs for layout generation and spatial reasoning [ 12 , 26 , 35 ] . However, these approaches assume that once a layout is specified, the T2I model will faithfully follow it, overlooking potential deviations during generation. Furthermore, their latent pasting approach struggles with occlusions, and disrupts the diffusion denoising process, as the pasted noise