Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models
Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim · 2025
Read Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models by Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08622.txt.
Introduction
Getting a text-to-image model to make exactly the picture in your head takes prompt craft, and working backward from a picture to its prompt is just as valuable. Earlier inversion methods produce soft prompts no human can read, or hard prompts that read as gibberish. The authors propose Visually Guided Decoding, which needs no gradients and no training. A large language model drafts readable prompt candidates in plain words. A CLIP model then scores each draft against the target visual idea, and the best-aligned wording wins.
Because the language model does the writing, the prompts stay coherent and meaningful to people. Experiments show the method beating earlier inversion techniques on how understandable and contextually fitting its prompts are. The authors argue that readable prompts restore control: users can edit, reuse, and learn from them. They present the approach as a more intuitive front door to image generation.
Catalog record
| CREATOR | Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim Abstract Text-to-image generative models like DALL-E and Stable Diffusion have revolutionized visual content creation across various applications, including advertising, personalized media, and design prototyping. However, crafting effective textual prompts to guide these models remains challenging, often requiring extensive trial and error. Existing prompt inversion approaches, such as soft and hard prompt techniques, are not so effective due to the limited interpretability and incoherent prompt generation. To address these issues, we propose Visually Guided Decoding (VGD), a gradient-free approach that leverages large language models (LLMs) and CLIP-based guidance to generate coherent and semantically aligned prompts. In essence, VGD utilizes the robust text generation capabilities of LLMs to produce human-readable prompts. Further, by employing CLIP scores to ensure alignment with user-specified visual concepts, VGD enhances the interpretability, generalization, and flexibility of prompt generation without the need for additional training. Our experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextually relevant prompts, facilitating more intuitive and controllable interactions with text-to-image models. (Implementation code is available at https://github.com/DonghoonKim-1938/VGD ) 1 Introduction Figure 1: Visually Guided Decoding ( VGD ) works with any LLM without extra training, making it easy to integrate into a chat-based interface that offers interpretable and controllable text-to-image generation. In recent years, image generative models such as DALL-E and Stable Diffusion have shown remarkable success in generating high-fidelity images ( Ramesh et al., 2022 ; Rombach et al., 2022 ; Podell et al., 2024 ) . These models are widely used in a variety of applications, including visual content generation ( e.g. , advertisement, movie, game), personalized content generation ( e.g. , caricature, photo editing), and prototyping ( e.g. , architecture and product design). Recent studies have shown that, just as humans can draw an image of an object solely based on detailed descriptions ( e.g. , criminal composite sketch), generative models can generate images of objects using a well-crafted prompt, even if they have not been trained on those specific objects ( Gal et al., 2023 ; Everaert et al., 2023 ) . Figure 2: VGD generates fully interpretable prompts that enhance generalizability across tasks and models in text-to-image generation. Figure 3: While conventional prompt inversion techniques update prompt embeddings through gradient-based optimization and quantization, VGD is a gradient-free technique that utilizes large language models and CLIP to generate relevant sentences. A well-known drawback of these approaches is the difficulty in finding a textual description ( a.k.a., prompt) that effectively guides the generation of the desired visual content ( Hao et al., 2024 ) . For example, to create a culinary masterpiece (see Fig. 1 ), one needs to have enough knowledge on exquisite cooking styles such as French cuisine or molecular gastronomy. Without the expertise to craft prompts, users have to rely on laborious trial and error to generate high-quality gourmet images. To address the difficulty in finding the prompt, soft prompt inversion techniques have been proposed ( Gal et al., 2023 ; Kumari et al., 2023 ; Ruiz et al., 2023 ; Voynov et al., 2023 ) . In these approaches, similar to the prompt learning technique in natural language processing ( Li & Liang, 2021 ; Lester et al., 2021 ; Liu et al., 2022 ; Kim et al., 2024 ) , an image is transformed into the embedding vector. Specifically, a learnable input vector for the conditioning network ( Radford et al., 2021 ) in Stable Diffusion is trained to reconstruct user-provided images through an iterative de-noising process ( a.k.a. , reverse diffusion process). After training, the learned vector is used as a soft prompt. Since the soft prompt is used as a word, to generate the desired image, it should be combined with other words (see Fig. 2 ). Unfortunately, it is a non-human-readable vector, users have no clue to handle it. Recently, to mitigate the lack of interpretability, hard prompt inversion has been suggested ( Mahajan et al., 2024 ; Wen et al., 2024 ) . Essentially, training process of the hard prompt inversion is similar to the soft-prompt learning but the trained input vector is projected to the nearest neighbor in a predefined vocabulary to ensure human-readability (see Fig. 3 (a)). The potential drawbacks of hard prompt inversion are: 1) the vanishing gradient problem caused by the long gradient path in the reverse diffusion process, and 2) the difficulty in identifying the essential words for generating the desired image. In fact, as described in Fig. 2 , a sequence of the hard prompts contain meaningless as well as useful words. An aim of this paper is to propose a novel prompt generation technique that generates fully interpretable prompts, thereby reducing the need for trial-and-error in text-to-image generation. Essence of our technique, henceforth referred to as visually guided decoding (VGD), is to generate contextually meaningful sentence for hard prompt through the token generation process of large language models (LLMs) ( Ouyang et al., 2022 ; Touvron et al., 2023 ) . In order to align the generated token in the provided images, we use the CLIP model in the token generation process (see Fig. 3 (b)). This process ensures that the resulting prompt is not only semantically relevant to the image but also easily understood by humans. The major benefit of VGD is that one can fully understand the meaning of the prompt ( interpretability ) and synthesize images for various applications ( generalizability ) ( e.g. , style transfer, image editing, and image variation). Also, when users want to modify objects, background, and/or style, one can easily change the generated prompt ( flexibility ). In our experiments, we show that VGD qualitatively and quantitatively achieves state-of-the-art (SOTA) performance, demonstrating superior interpretability, generalizability, and flexibility in text-to-image generation compared to previous soft and hard prompt inversion methods. We also show that VGD is compatible with a combination of various LLMs ( i.e. , LLaMA2, LLaMA3, Mistral) and image generation models ( i.e. , DALL-E 2, MidJourney, Stable Diffusion 2). Since VGD seamlessly stitches LLMs and text-to-image models without additional training ( i.e. , gradient-free), VGD provides a flexible and efficient solution for chat-interfaced image generation services using LLM ( e.g. , DALL-E extension on ChatGPT) (see Fig. 1 ). 2 Background and Related Works 2.1 CLIP Model CLIP model aligns semantically related visual and textual content within a shared representation space ( Radford et al., 2021 ; Lee et al., 2025 ) . It consists of an image encoder \text{CLIP}_{\text{img}}(\cdot) and a text encoder \text{CLIP}_{\text{txt}}(\cdot) . The image encoder encodes an input image I into a visual embedding \mathbf{f}_{\text{img}}=\text{CLIP}_{\text{img}}(I) . The text encoder processes the input text T=\left[x_{1}^{\text{txt}},x_{2}^{\text{txt}},...,x_{N}^{\text{txt}}\right] to generate a textual embedding \mathbf{f}_{\text{txt}}=\text{CLIP}_{\text{txt}}(T) . CLIP is trained using a contrastive learning approach such that the similarity between \mathbf{f}_{\text{txt}} and \mathbf{f}_{\text{img}} for matched sentence-image pair is maximized, while the similarity for unmatched pairs is minimized. The similarity is defined as: \text{Sim}(\mathbf{f}_{\text{txt}},\mathbf{f}_{\text{img}})\coloneq s\frac{\mathbf{f}_{\text{txt}}\cdot\mathbf{f}_{\text{img}}}{\|\mathbf{f}_{\text{txt}}\|_{2}\|\mathbf{f}_{\text{img}}\|_{2}} (1) where s is a scalar scaling factor. Then, the training objective of CLIP is formulated as: \underset{\text{CLIP}_{\text{img}},\text{CLIP}_{\text{txt}}}{\mathrm{\textbf{Maximize}}}\quad\log P\left(T|I\right)+\log P\left(I|T\right), (2) P(T|I)=\frac{\exp\left(\text{Sim}({{\mathbf{f}}_{\text{txt}}}\cdot{\mathbf{f}}_{\text{img}})\right)}{\sum_{j=1}^{B}\exp\left(\text{Sim}({{\mathbf{f}}_{\text{txt}_{j}}}\cdot{\mathbf{f}}_{\text{img}})\right)},\quad P(I|T)=\frac{\exp\left(\text{Sim}({{\mathbf{f}}_{\text{txt}}\cdot{\mathbf{f}}_{\text{img}}})\right)}{\sum_{j=1}^{B}\exp\left(\text{Sim}({{\mathbf{f}}_{\text{txt}}\cdot{\mathbf{f}}_{\text{img}_{j}}})\right)}, (3) where B indicates the number of text-image pairs in the mini-batch. 2.2 Text-to-Image Diffusion Model