Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval
Mingyu Liu Sihan Huang Yijia Fan Yinlin Yan Quan Zhang, [0.05em] Jian-Fang Hu Jianhuang Lai, [0.12em] Sun Yat-sen University, [-0.02em] Guangdong Province Key Laboratory of Information Security Technology, China, [-0.02em] Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China · 2026
Read Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval by Mingyu Liu Sihan Huang Yijia Fan Yinlin Yan Quan Zhang, [0.05em] Jian-Fang Hu Jianhuang Lai, [0.12em] Sun Yat-sen University, [-0.02em] Guangdong Province Key Laboratory of Information Security Technology, China, [-0.02em] Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08389.txt.
Introduction
Mingyu Liu, Sihan Huang, Yijia Fan, Yinlin Yan, and coauthors fix a subtle flaw in zero-shot composed image retrieval. A user supplies a photo plus a text edit. The system must find the edited target. No labeled training triplets exist. Lightweight projection methods are cheap. But they treat the edit text as a label for the target image. They miss that it is a change applied to the source.
The authors show that one adapter trained for both jobs creates a conflict. Endpoint matching fights transition learning. DeCIR separates them. Paired forward and reverse edit tuples are built from image-caption pairs. They train distinct low-rank modules for endpoints and for semantic transitions. The bottleneck goes away.
Catalog record
| CREATOR | Mingyu Liu Sihan Huang Yijia Fan Yinlin Yan Quan Zhang, [0.05em] Jian-Fang Hu Jianhuang Lai, [0.12em] Sun Yat-sen University, [-0.02em] Guangdong Province Key Laboratory of Information Security Technology, China, [-0.02em] Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08389
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval Mingyu Liu Sihan Huang Yijia Fan Yinlin Yan Quan Zhang, [0.05em] Jian-Fang Hu Jianhuang Lai, [0.12em] Sun Yat-sen University, [-0.02em] Guangdong Province Key Laboratory of Information Security Technology, China, [-0.02em] Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China Abstract Zero-shot composed image retrieval (ZS-CIR) retrieves a target image from a reference image and a text modification without human-annotated CIR triplets. Projection-based ZS-CIR methods are attractive because they do not rely on LLMs at inference and remain lightweight, but they often underperform LLM-based approaches on complex semantic modifications. This gap reflects a semantic transition bottleneck in projection-based ZS-CIR: endpoint-level matching can let the edit text act as a target-side attribute cue rather than grounding it as a source-conditioned semantic transition. We further show that adding semantic transition supervision to the same text adapter creates an endpoint–transition conflict between endpoint alignment and semantic transition alignment. To address this conflict, DeCIR decouples endpoint and transition learning. It constructs paired forward/reverse edit tuples from image-caption pairs, trains separate low-rank text adapter branches for endpoint alignment and semantic transition alignment, and merges them with Low-Rank Directional Merge (LRDM) into one deployable adapter. Extensive experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity. 1 Introduction Figure 1: Illustration of our motivation. The semantic transition bottleneck in projection-based ZS-CIR can lead to an endpoint shortcut, where the edit text is treated as a target-side cue while source-conditioned evidence from the reference is dropped. This motivates DeCIR to explicitly learn transition-aware matching. Composed image retrieval (CIR) searches for a target image that matches both a reference image and a text modification. The target should apply the requested edit while preserving relevant visual evidence from the reference, i.e., it must be reachable from the reference under the instruction rather than merely satisfy an endpoint cue. Conventional CIR models learn this behavior from human-annotated reference–instruction–target triplets, which are expensive to collect and difficult to scale. Zero-shot composed image retrieval (ZS-CIR) instead trains on large image-caption datasets such as CC3M [ 34 ] , but without triplets the model must infer source-to-target semantic transitions from weaker supervision. This makes the central supervision problem sharper: the model must infer how the source should change, even though the desired transition is not directly observed. Existing ZS-CIR methods mainly follow two routes. LLM-based methods leverage the knowledge and reasoning abilities acquired during pretraining to infer a semantic transition from the source to the target [ 18 , 36 , 13 , 40 , 42 ] . By rewriting the composed query into a target caption, LLMs serve as a bridge from the reference–modification input to a hypothesized target endpoint, making them effective at handling complex edits, but often at the cost of heavier inference pipelines. Projection-based methods take a much lighter route: they map the reference image into the text embedding space of a pretrained vision-language model such as CLIP, combine it with the modification text, and encode the resulting prompt into a composed query embedding for retrieval [ 33 , 1 , 39 , 37 , 50 ] . This route is attractive for deployment because it avoids heavy LLM calls and keeps the retrieval pipeline lightweight. However, by avoiding an explicit reasoning bridge, projection-based ZS-CIR leaves the source-to-target transition to be inferred implicitly by the CLIP text encoder, which encodes the pseudo-word and modification into a retrieval embedding rather than explicitly constructing target semantics as an LLM can. This raises a key question: can such a lightweight projection-based pipeline reliably capture source-to-target semantic transitions for complex modifications? We argue that existing projection-based methods remain limited by a semantic transition bottleneck : The CLIP text encoder is expected to infer the source-to-target semantic transition, although such transitions are neither explicitly supervised in contrastive pretraining nor directly optimized during pseudo-word mapping network training. Without source-conditioned semantic transition supervision, the modification text can collapse into a target-side attribute cue under complex edits: the model may capture the target attribute in the instruction without binding it to the source condition provided by the reference image. For example, as shown in Figure 1 , the reference image supplies the source-conditioned cue, a green bird, while the instruction is only “make it two birds”. A projection-based model may treat the text as the endpoint cue “two birds” and retrieve any image containing two birds, instead of learning the semantic transition from one green bird in the reference image to two green birds. This shortcut can work in easy cases but becomes brittle when hard distractors share the same target attribute without satisfying the intended source-conditioned semantic transition. Explicit semantic transition supervision is a natural way to address this bottleneck. However, we further show that adding it to the same adapter introduces a new optimization challenge. Endpoint alignment pulls the composed query toward target endpoints, whereas semantic transition alignment asks instruction embeddings to represent source-to-target displacement. These objectives are complementary for CIR, but forcing them through the same low-rank text adapter can cause gradient interference, which is especially pronounced in deeper text transformer blocks and thus degrade retrieval performance. This evidence motivates conflict-aware endpoint–transition decoupling: the model should allow endpoint alignment and transition alignment to specialize during training while keeping a lightweight projection-based retrieval pipeline at inference. We propose DeCIR, a conflict-aware endpoint–transition decoupling framework for lightweight ZS-CIR. We leverage an LLM to convert image-caption pairs into paired forward/reverse edit tuples, providing diverse semantic transition supervision without human-annotated CIR triplets. DeCIR then trains two low-rank text-adapter branches over the CLIP text encoder: an endpoint alignment branch that aligns the composed query embedding with the target-caption endpoint, and a transition branch for instruction-level semantic displacement. Finally, Low-Rank Directional Merge (LRDM) folds the semantic transition alignment branch into the endpoint alignment branch, producing a single deployable text adapter. The resulting model preserves the standard projection-based inference pipeline while obtaining stronger source-conditioned transition awareness. Overall, our contributions are summarized as follows: \bullet We identify a semantic transition bottleneck in projection-based ZS-CIR, where edit text is matched as a target-side cue rather than grounded as a source-conditioned transition. \bullet We show that endpoint and transition objectives interfere in the same low-rank adapter, motivating decoupled endpoint and transition learning. \bullet We introduce LRDM, a shared-basis coefficient merge that folds transition sensitivity into the projection-based retrieval pipeline while retaining lightweight inference. \bullet Experiments on CIRR, CIRCO, FashionIQ, and GeneCIS show that DeCIR improves projection-based ZS-CIR without LLMs or reasoning loops at inference. 2 Related Work 2.1 Zero-Shot Composed Image Retrieval Composed image retrieval was first studied with human-annotated reference–instruction–target triplets [ 41 , 5 , 44 , 28 , 2 ] . ZS-CIR removes these triplets and mainly follows two paths. Projection-based methods map the reference image into the text space and retrieve with a single composed query [ 33 , 1 , 39 , 37 , 50 ] ; related CIR work further improves composition modules, diffusion proxies, hard negatives, and robustness settings [ 3 , 8 , 20 , 31 , 35 ] . These methods strengthen different parts of the endpoint-matching pipeline, such as source pseudo-word inversion, context-dependent