Thinking Before Retrieving: Robust Zero-Shot Composed Image Retrieval via Strategic Planning and Self-Criticism Journal: Pattern Recognition

Gunho Jung, Jeong-Woo Park, Seon Bin Kim, Seong-Whan Lee · 2026

Read Thinking Before Retrieving: Robust Zero-Shot Composed Image Retrieval via Strategic Planning and Self-Criticism Journal: Pattern Recognition by Gunho Jung, Jeong-Woo Park, Seon Bin Kim, Seong-Whan Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31222.txt.

Introduction

Jung and three colleagues improve composed image retrieval, finding a target picture from a reference image plus a written change. In the zero-shot setting, a frozen vision language space is used and the query text is built at test time. Single-pass query writing can garble details or drop required changes without notice.

Their PEC-CIR framework splits the job among a Planner, an Executor, and a Critic. The Planner lists explicit constraints, the Executor drafts several candidate descriptions, and the Critic checks each draft against the constraints before retrieval. Staging the work this way cuts error spread and steadies results. The study frames careful query building, not extra training, as the lever for more robust retrieval.

Catalog record

CREATORGunho Jung, Jeong-Woo Park, Seon Bin Kim, Seong-Whan Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31222

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thinking Before Retrieving: Robust Zero-Shot Composed Image Retrieval via Strategic Planning and Self-Criticism Journal: Pattern Recognition Gunho Jung, Jeong-Woo Park, Seon Bin Kim, Seong-Whan Lee Abstract Composed image retrieval requires identifying a target image from a gallery by integrating a reference image with a textual modification instruction. In a training-free zero-shot setting, this task relies on constructing a retrieval-oriented textual query within a frozen vision–language embedding space at inference time. Existing approaches predominantly rely on a single-pass generation strategy that fuses the reference context and modification text into a unified description. This strategy makes it difficult to detect or correct semantic distortions and omissions during generation. Consequently, the preservation of reference attributes and the integration of textual requirements interfere with each other, which degrades retrieval precision. To address these challenges, we introduce PEC-CIR, a training-free framework that structures query construction as a multi-stage reasoning pipeline. The framework operates through a Planner–Executor–Critic architecture where the Planner extracts explicit constraints, the Executor generates multiple candidate target descriptions, and the Critic evaluates these candidates based on constraint compliance. By reframing query construction as a staged inference process instead of a single-pass output, PEC-CIR reduces the propagation of generative errors by explicitly evaluating candidate queries before retrieval, thereby improving retrieval stability. 1 Introduction Composed Image Retrieval (CIR) aims to retrieve a target image by synergizing a reference image with a textual instruction [ 42 , 29 , 12 , 22 , 36 ] . Unlike conventional retrieval based solely on visual similarity, this task demands the preservation of specific visual attributes while simultaneously modifying features in accordance with a textual prompt [ 43 , 49 ] . For example, in e-commerce, a query such as “the same design but in a different color without a logo” exemplifies the need for precise multimodal integration. This cross-modal synergy enables the expression of fine-grained preferences, allowing for the retrieval of images that satisfy nuanced modifications while maintaining visual identity [ 16 , 9 , 15 ] . Compositional retrieval requirements arise across various application domains, including fashion, interior design, and personalized recommendation, highlighting the practical relevance of CIR in real-world scenarios. Figure 1: Comparison between a typical training-free CIR approach and the proposed PEC-CIR. Early CIR studies relied on supervised learning with annotated triplets consisting of a reference image, a modification text, and a target image [ 3 , 55 ] . While achieving strong performance through joint visual-textual representations, supervised frameworks require large-scale annotations and incur substantial computational costs. To minimize supervision overhead, recent work explores zero-shot CIR, leveraging pre-trained vision-language models to perform multimodal fusion without task-specific training [ 1 , 5 , 33 ] . Notably, the integration of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) emerges as a prominent paradigm owing to their advanced reasoning capabilities for translating complex multimodal instructions into descriptive target representations. The generative strategy produces a natural language description of the target image for subsequent text-to-image retrieval, facilitating a flexible and training-free inference process [ 28 , 40 , 52 , 41 , 6 ] . However, existing generative zero-shot frameworks suffer from three structural limitations. First, the decoupled processing of the reference image and textual instruction limits explicit cross-modal interaction between the visual context and the editing intent. The separation creates ambiguity regarding the preservation or modification of specific visual attributes, resulting in inaccurate retrieval results. Furthermore, the stochastic nature of text generation introduces semantic volatility during query construction. Probabilistic decoding leads to the omission of critical visual constraints or the introduction of extraneous concepts, thereby undermining retrieval consistency. Finally, previous paradigms uncritically adopt generated outputs as optimal, overlooking the inherent volatility in retrieval quality across diverse candidates. This absence of discriminative oversight allows hallucinations or semantic misalignments to propagate unchecked to the retrieval engine, compromising the robustness of training-free frameworks. Figure 1 (a) illustrates a representative failure case of an existing training-free CIR approach, in which a single-pass generated query captures only partial aspects of the textual instruction, yielding misaligned retrieval results. To address these challenges, we propose PEC-CIR, a robust training-free framework that redefines zero-shot query construction as a structured reasoning pipeline. Our framework decomposes the process into three interconnected stages: Planning, Execution, and Criticism. The Planner first extracts structured constraints for preservation and modification, grounding the reasoning in visual context. The Executor then synthesizes diverse candidate descriptions to mitigate the volatility of stochastic generation. Finally, the Critic evaluates these candidates against objective criteria, ensuring that only descriptions satisfying strict retrieval requirements are propagated. The multi-stage design stabilizes query construction, yielding consistent performance gains on the FashionIQ [ 47 ] and CIRR [ 26 ] benchmarks. Our contributions are summarized as follows: 1. We propose PEC-CIR, a training-free framework that redefines zero-shot CIR as a multi-stage reasoning pipeline, effectively mitigating the volatility inherent in single-pass generative models. 2. We introduce an agentic refinement mechanism facilitated by a feedback-driven loop, which prevents the propagation of hallucinations through rigorous visual-textual verification. 3. Experimental results on FashionIQ and CIRR benchmarks demonstrate that PEC-CIR achieves consistent performance gains, achieving state-of-the-art performance among training-free compositional retrieval methods. 2 Related Works 2.1 Composed Image Retrieval Composed Image Retrieval (CIR) aims to retrieve a target image from a gallery by integrating a reference image with a textual modification instruction [ 42 , 29 , 12 , 27 , 54 ] . The fundamental difficulty entails identifying which visual attributes of the reference image require preservation and which necessitate modification according to the instruction. The dual requirements of preservation and modification represent the expression of complex editing intents and necessitate an intricate interaction between visual and textual modalities. Early studies on CIR focused on supervised learning setting, using annotated triplets consisting of a reference image, a modification text, and labeled targets for training [ 46 , 50 ] . More broadly, supervised cross-modal retrieval has also explored modality-invariant representation learning and semantic relation modeling through label-aware graph convolution, dual adversarial graph neural networks, and multi-label contrastive learning [ 31 , 32 , 30 ] . While effective within the distribution of annotated triplets, supervised designs inherently depend on large-scale supervision. Consequently, static architectures lack the flexibility to accommodate novel attributes or compositions without the significant overhead of additional data collection and model retraining [ 5 ] . Related zero-shot cross-modal retrieval tasks have also been studied in zero-shot sketch-based image retrieval (ZS-SBIR), where models must align sketches and photos while generalizing to unseen categories. Recent ZS-SBIR studies address this challenge through relation-aware meta-learning, bidirectional knowledge distillation, and mixed-sample augmentation with bidirectional mining [ 24 , 10 , 25 ] . 2.2 Zero-Shot Composed Image Retrieval 2.2.1 Representation-based approach Representation-based frameworks focus on composing image and instruction features directly within the embedding space of pre-trained vision–language models [ 2 , 17 , 48 , 18 ] . In vector-fusion paradigms, the reference image and the modification text undergo separate encoding before combining via weighted fusion [ 17 ] , linear transformations [ 4 ] , or similarity-based operations [ 14 ] . Direct vector composition is computationally efficient and requires minimal task-specific training. However, the reliance on pre-defined feature spaces limits the ability to represent complex semantics, particularly under conditional modifications where

Read the complete text