GenIR: Generative Visual Feedback for Mental Image Retrieval

Diji Yang, Minghao Liu, Chung-Hsiang Lo, Yi Zhang, James Davis · 2025

Read GenIR: Generative Visual Feedback for Mental Image Retrieval by Diji Yang, Minghao Liu, Chung-Hsiang Lo, Yi Zhang, James Davis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06220.txt.

Introduction

Diji Yang and co-authors study image search where the target lives only in the user's mind, in a 2025 paper. Real searches unfold over rounds, guided by a mental picture that may be vague or vivid. Text feedback from the machine is often too abstract to steer by, though.

Their system, GenIR, answers each round with a generated image that shows what the machine thinks the user wants. Seeing that picture, users can correct the search in plain, visual terms. The team also built an automated way to make multi-round test data for this task. Trials show the visual feedback loop finds target images better than word-based loops. The work reframes search as a shared sketching session between user and model.

Catalog record

CREATORDiji Yang, Minghao Liu, Chung-Hsiang Lo, Yi Zhang, James Davis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06220

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GenIR: Generative Visual Feedback for Mental Image Retrieval Diji Yang, Minghao Liu, Chung-Hsiang Lo, Yi Zhang, James Davis Abstract Vision-language models (VLMs) have shown strong performance on text-to-image retrieval benchmarks. However, bridging this success to real-world applications remains a challenge. In practice, human search behavior is rarely a one-shot action. Instead, it is often a multi-round process guided by clues in mind. That is, a mental image ranging from vague recollections to vivid mental representations of the target image. Motivated by this gap, we study the task of Mental Image Retrieval (MIR), which targets the realistic yet underexplored setting where users refine their search for a mentally envisioned image through multi-round interactions with an image search engine. Central to successful interactive retrieval is the capability of machines to provide users with clear, actionable feedback; however, existing methods rely on indirect or abstract verbal feedback, which can be ambiguous, misleading, or ineffective for users to refine the query. To overcome this, we propose GenIR, a generative multi-round retrieval paradigm leveraging diffusion-based image generation to explicitly reify the AI system’s understanding at each round. These synthetic visual representations provide clear, interpretable feedback, enabling users to refine their queries intuitively and effectively. We further introduce a fully automated pipeline to generate a high-quality multi-round MIR dataset. Experimental results demonstrate that GenIR significantly outperforms existing interactive methods in the MIR scenario. This work establishes a new task with a dataset and an effective generative retrieval method, providing a foundation for future research in this direction 1 1 1 Code and data are available at https://github.com/mikelmh025/generative_ir . . 1 Introduction Figure 1: Comparison of methods for Mental Image Retrieval task. Top : our generative method, which reifies the intermediate query using an image generator model and applies image-to-image search for retrieval. Bottom : Existing approach (ChatIR and PlugIR) which support multi-round query improvements based on verbal feedback. Recent Vision-language models (VLMs) have achieved decent results on standard text-to-image retrieval benchmarks wang2024qwen2 ; li2024llava . Despite this progress, transferring these capabilities into real-world applications remains challenging. One key limitation is that real human search behavior is often not one-shot or static; it unfolds through a sequence of actions, highlighting the necessity for interactive information retrieval (IIR) systems rui1999image ; white2009exploratory ; borlund2013interactive . Another limitation is that users frequently initiate a search to re-find the information they have seen before, which could be partial memory, vague clues, or vivid recall of the target images dumais2003stuff ; zhou2003relevance ; guo2018dialog . To address this scenario, we define Mental Image Retrieval (MIR) 2 2 2 The term is inspired by the Mental Image Reconstruction task koide2024mental and shares the definition of “mental image”, though their study purely from Neuroscience side which is different problem from ours. , where users iteratively refine their queries based on mental image (i.e., an image in mind) to retrieve an intended image from an image database. Although MIR has not been explicitly formulated as a distinct task previously, as a subset of long-standing text-to-image IIR task thomee2012interactive ; guo2018dialog , some works have implicitly touched upon similar settings. ChatIR levy2023chatting positions a VLM (or ideally a human) as the active searcher. Seeing the initial query from the searcher, ChatIR uses Large Language Models (LLMs) to provide system feedback in question format to the user solely based on the textual dialogue history. The question is then answered by the searcher who has access to the ground-truth images (i.e., Mental Image). Next, the dialogue history will be appended with the question-answer pair and then be used as the search query. This implicitly positions ChatIR within the realm of MIR as a subset of interactive text-to-image retrieval where the human searcher holds the target image in memory. However, as shown in the bottom left of Figure 1 , ChatIR has only verbal (text-based) feedback with no information from image space, resulting in generated question-answer pairs may be redundant or irrelevant to the query refinement. PlugIR lee2024interactive , as shown in the bottom right of Figure 1 , advances this setup further by incorporating retrieval context—text captions of retrieved images into the query generation for subsequent rounds, aiming to produce more contextually relevant feedback and mitigate redundancy. Nevertheless, the major challenge exists, both methods remain constrained by significant limitations regarding feedback effectiveness yang2024tackling . Even across multiple interaction rounds, these methods rely heavily on indirect, verbal feedback derived solely from retrieval failures. Such feedback is often abstract and interpretability-poor, providing users with little actionable insight or potentially misleading clues for refining subsequent queries. In vision-language embedding spaces, such as CLIP radford2021learning , minor textual edits can cause unpredictable changes in retrieval outcomes, making query refinement inherently a trial-and-error process. Consequently, the feedback offered by existing conversational retrieval approaches inadequately expresses the AI system’s current understanding and fails to directly benefits users toward effective refinements. As ChatIR example shown in Figure 6 , the verbal feedback for an image depicting a person wearing a motorcycle helmet: “a human is not wearing a hat”. Although literally true, such feedback fails to capture the visual salience of the helmet and may steer the user’s refinement toward irrelevant details, ultimately misleading the search process for images containing headwear. Motivated by the need for more effective and interpretable system feedback, we propose GenIR, a generative interactive retrieval paradigm designed explicitly to provide clear, interpretable, and actionable visual feedback at each interaction turn. GenIR employs a straightforward but powerful iterative pipeline as shown in Figure 1 : first, a text-to-image diffusion model generates a synthetic image from the user’s current textual query; then, this synthetic image is used for retrieval from a database through image-to-image similarity matching. Crucially, the generated synthetic image serves as more than just a query, but acts as an explicit visualization of the system’s internal understanding (i.e., the representation of the query in the vision-language latent space), enabling users to clearly perceive discrepancies between their mental image and the system’s interpretation so that can refine the query for the next round search. Beyond its utility at inference time, GenIR also supports dataset construction for studying MIR. Following the common practice of using VLM to play as the human searcher levy2023chatting ; lee2024interactive , we create an automated pipeline based on the GenIR framework. We present a multi-round dataset with each round consisting of a refined query, a generated synthetic image, and retrieved results with a correctness label. Our experiments demonstrate that GenIR outperforms exisitng MIR baselines, highlighting the significant advantage of using visual feedback over verbal feedback. Furthermore, our study indicates that our GenIR annotated query can result in better retrieval performance than annotation from ChatIR under the same retrieval setting. In summary, our contribution is as follows: • Task: We formally define the Mental Image Retrieval (MIR) task, a subset of the multi-round interactive text-to-image retrieval task where the searcher has the target image in mind. • Method: We propose a novel framework, GenIR, using the generative approach to provide intuitive, interpretable visual feedback revealing the optimization direction for user during multi-round query refinement. • Dataset: We release a dataset with visually grounded feedback annotation, together with a curation pipeline which can support both MIR and general text-to-image

Read the complete text