Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions

Jihoon KwonSeoul National University [email protected], Kyle MinOracle [email protected], Jy-yong SohnYonsei University [email protected] · 2025

Read Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions by Jihoon KwonSeoul National University [email protected], Kyle MinOracle [email protected], Jy-yong SohnYonsei University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16540.txt.

Introduction

Jihoon Kwon, Kyle Min, and Jy-yong Sohn boost a vision-language model in their 2025 paper. Such models link images and text, but they grasp word order and links poorly. A model may know the words cup and table yet miss which sits on which. Their method, READ, fine-tunes the CLIP model with two extra goals.

One goal makes a frozen text tool rebuild a new caption from the old one's code. The other pulls two wordings of one sense close in code space. READ-CLIP set the best scores on five tests of this skill. Gains over the best plain fine-tune ran up to about four percent. Both extra goals help, each in its own way.

Catalog record

CREATORJihoon KwonSeoul National University [email protected], Kyle MinOracle [email protected], Jy-yong SohnYonsei University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16540

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions Jihoon KwonSeoul National University [email protected], Kyle MinOracle [email protected], Jy-yong SohnYonsei University [email protected] Abstract Despite recent advances, vision-language models trained with standard contrastive objectives still struggle with compositional reasoning – the ability to understand structured relationships between visual and linguistic elements. This shortcoming is largely due to the tendency of the text encoder to focus on individual words rather than their relations, a limitation reinforced by contrastive training that primarily aligns words with visual objects. In this paper, we introduce REconstruction and Alignment of text Descriptions (READ), a fine-tuning method designed to enhance compositional reasoning by adding two auxiliary objectives to the contrastive learning: (1) a token-level reconstruction objective, where a frozen pre-trained decoder reconstructs alternative captions based on the embedding of the original caption; and (2) a sentence-level alignment objective, which explicitly aligns paraphrased sentences in the embedding space. We show that READ-CLIP, a model derived by applying the READ method to the pre-trained CLIP model, achieves the state-of-the-art performance across five major compositional reasoning benchmarks, outperforming the strongest conventional fine-tuning baseline by up to 4.1%. Furthermore, applying the READ to existing CLIP variants (including NegCLIP and FSC-CLIP) also improves performance on these benchmarks. Quantitative and qualitative analyses reveal that our proposed objectives – reconstruction and alignment – offer complementary benefits: the former encourages the encoder to capture relationships between words within a caption, while the latter ensures consistent representations for paraphrases expressed with different wording. 1 Introduction Recent advances in Vision-Language Models (VLMs) have significantly enhanced the ability to align images with text descriptions [ 8 , 67 ] . A key driver of this progress is contrastive pre-training, such as CLIP [ 45 ] , which learns to embed images and texts into a shared multi-modal space, in a way that the distance in the embedding space represents the semantic similarity of image-text pairs. VLMs trained with this standard contrastive objective have been widely applied to diverse downstream tasks, including open-vocabulary object detection [ 17 , 72 ] , semantic segmentation [ 15 , 30 , 34 ] , cross-modal retrieval [ 12 , 36 , 71 ] , and multi-modal generation [ 1 , 47 , 53 ] . Despite their remarkable progress, current VLMs still face challenges with compositional reasoning – the ability to understand structured relationships between visual and linguistic elements [ 37 , 56 , 64 ] . Numerous studies have shown that VLMs commonly fail on even simple compositional tasks that humans find straightforward [ 11 , 19 , 24 , 28 , 40 , 42 , 58 , 66 , 69 ] . For instance, when given an image of a horse eating grass, VLMs often assign a higher similarity score to the incorrect caption “the grass is eating the horse” than to the correct caption “the horse is eating the grass”, highlighting the limitation of the VLMs in capturing syntactic and relational structures [ 64 ] . These failures underscore the need for further research on compositional reasoning to achieve reliable and robust vision-language understanding in real-world applications [ 4 , 9 , 29 , 33 , 39 , 56 , 57 , 60 ] . Figure 1 : Overview of the training objectives used in READ-CLIP (left) , a VLM that applies REconstruction and Alignment of text Descriptions (READ) method to the pretrained CLIP model [ 45 ] . READ is our proposed fine-tuning method that enhances compositional reasoning in VLMs by augmenting contrastive learning with two auxiliary objectives. The auxiliary objectives consist of two components: token-level reconstruction and sentence-level alignment . The performance of READ-CLIP (right) on compositional reasoning benchmarks demonstrates that it consistently outperforms conventional state-of-the-art methods across diverse aspects of compositional reasoning. Prior works have identified the text encoder as a primary bottleneck for compositional reasoning in VLMs trained with the contrastive objective. Specifically, the text encoder often fails to capture the relationship between words in a sentence [ 10 , 23 , 65 ] . This limitation is largely attributed to the contrastive objective, which trains the text encoder to align a caption with its corresponding image, without encouraging the encoder to capture the relationships between words [ 10 , 64 ] . As a result, the text encoder focuses on words that refer to objects depicted in the image, as they mainly contribute to the image-text alignment, thereby limiting its ability to learn compositional reasoning [ 7 ] . Conventional approaches to overcome this limitation can be categorized into two parts. The first approach is to modify the contrastive objective by introducing hard negatives [ 48 ] – semantically different examples that are nonetheless difficult for the model to distinguish from the positives. Here, the model is trained to bring each sample closer to its positives and push it farther from hard negatives, thus improving the compositional reasoning capability [ 41 , 50 , 54 , 64 ] . However, these approaches that rely solely on hard negatives often encourage the model to focus on patterns specific to those negatives, rather than developing genuine compositional reasoning [ 13 , 14 , 19 ] . The second approach is to add auxiliary objectives to the standard contrastive objective. However, these efforts either supervise both image and text encoders jointly [ 51 , 70 ] or focus solely on the image encoder [ 2 , 18 ] . Although the text encoder serves as a primary bottleneck for compositional reasoning, limited attention has been paid to adopting auxiliary objectives for the text encoder, aimed at improving the compositional reasoning capability. In order to enhance the compositional reasoning capability of VLMs more effectively, we tackle the underexplored challenge of improving the text encoder via a targeted training objective. Specifically, our contributions are as follows: • In Sec. 3 , we propose REconstruction and Alignment of text Descriptions (READ), a fine-tuning method designed to enhance the text encoder by adding two auxiliary objectives to the standard contrastive objective: (1) token-level reconstruction and (2) sentence-level alignment, as in Fig. 1 . First, the token-level reconstruction objective trains the text encoder to produce embeddings from the original caption that enable a frozen decoder to reconstruct each token of an alternative caption. Second, the sentence-level alignment objective explicitly aligns paraphrased captions in the embedding space to reflect their shared semantics, even when they are expressed differently. • In Sec. 4 , we provide experiments demonstrating the READ method is effective across a wide range of compositional reasoning benchmarks. Specifically, we introduce READ-CLIP, a VLM derived by applying the READ method to the pre-trained CLIP model [ 45 ] , which achieves the state-of-the-art performance on five compositional reasoning benchmarks. READ-CLIP outperforms the famous baseline NegCLIP [ 64 ] by an average of 4.5% across benchmarks, and outperforms the strongest baseline FSC-CLIP [ 38 ] by up to 4.1%. Furthermore, applying the READ method to existing CLIP variants (including NegCLIP and FSC-CLIP) consistently improves performance across these benchmarks, with gains of up to 2.4%. • Our analysis in Sec. 5 demonstrates that the two objectives in the READ method – reconstruction and alignment – provide complementary benefits for compositional reasoning. The

Read the complete text