Decoupled Global-Local Alignment for Improving Compositional Understanding
Xiaoxing Hu, Kaicheng Yang, Jun Wang, Haoran Xu, Ziyong Feng, Yupei Wang · 2025
Read Decoupled Global-Local Alignment for Improving Compositional Understanding by Xiaoxing Hu, Kaicheng Yang, Jun Wang, Haoran Xu, Ziyong Feng, Yupei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16801.txt.
Introduction
Xiaoxing Hu and co-authors present a 2025 paper on image-text models. CLIP joins images and text by global matching and does well on many tasks. Global matching is weak on composition: relations and attributes in a scene. Past fixes used hard negative texts to teach finer splits. Those fixes hurt the model's general skill by pushing negatives too far away. The team splits the fix into global and local tracks.
Their frame, called DeGLA, trains the two tracks apart. A self-distillation step inside the global track guards the model's base skill. Local training then sharpens relation and attribute grasp. Vision-language teams get finer scene reading without the usual skill tax. That trade-off has blocked past work, so a clean split is welcome. Tests in the paper check both composition scores and general scores side by side.
Catalog record
| CREATOR | Xiaoxing Hu, Kaicheng Yang, Jun Wang, Haoran Xu, Ziyong Feng, Yupei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decoupled Global-Local Alignment for Improving Compositional Understanding Xiaoxing Hu, Kaicheng Yang, Jun Wang, Haoran Xu, Ziyong Feng, Yupei Wang Abstract . Contrastive Language-Image Pre-training (CLIP) has achieved success on multiple downstream tasks by aligning image and text modalities. However, the nature of global contrastive learning limits CLIP’s ability to comprehend compositional concepts, such as relations and attributes. Although recent studies employ global hard negative samples to improve compositional understanding, these methods significantly compromise the model’s inherent general capabilities by forcibly distancing textual negative samples from images in the embedding space. To overcome this limitation, we introduce a De coupled G lobal- L ocal A lignment ( DeGLA ) framework that improves compositional understanding while substantially mitigating losses in general capabilities. To optimize the retention of the model’s inherent capabilities, we incorporate a self-distillation mechanism within the global alignment process, aligning the learnable image-text encoder with a frozen teacher model derived from an exponential moving average. Under the constraint of self-distillation, it effectively mitigates the catastrophic forgetting of pretrained knowledge during fine-tuning. To improve compositional understanding, we first leverage the in-context learning capability of Large Language Models (LLMs) to construct about 2M high-quality negative captions across five types. Subsequently, we propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC) loss to enhance vision-language compositionally. Experimental results across both general and compositional reasoning tasks validate the effectiveness of the DeGLA framework. Our code is released at https://github.com/xiaoxing2001/DeGLA . Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755032 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Computer vision representations (a) General performance. (b) Compositional performance (c) An illustration for the compositional understanding. Figure 1 . (a) General performance comparison across 11 classification datasets. (b) Compositional performance comparison on VALSE, ARO, and SugarCrepe benchmarks. (c) CLIP scores for images with aligned and unaligned captions. DeGLA demonstrates significantly enhanced capabilities in compositional understanding. 1. Introduction The rapid expansion of mobile networks and social platforms significantly boosts the large-scale generation of image-text pairs, laying a crucial foundation for vision-language pre-training ( Gu et al., 2025b ; Gu et al., 2025a ) . In particular, Contrastive Language-Image Pre-training ( Radford et al., 2021a ) (CLIP) leverages two distinct unimodal encoders for images and texts, and employs contrastive loss ( Oord et al., 2018a ) for representation learning. After pretraining on extensive image-text pairs sourced from the internet, CLIP exhibits strong transferability and is extensively applied across various tasks, including image captioning ( Alayrac et al., 2022 ; Li et al., 2023b ; Mañas et al., 2022 ) , object detection ( Kaul et al., 2023 ; Ma et al., 2023 ; Wu et al., 2024b ) , and segmentation ( Li et al., 2022b ; Cho et al., 2024 ; Yu et al., 2023 ; Minderer et al., 2022 ; Hoyer et al., 2025 ; Gu et al., 2021 ; Zang et al., 2022 ; Wu et al., 2024a ) . As CLIP attracts increasing attention from researchers, several enhanced methods based on CLIP have been proposed ( Li et al., 2023a ; Gu et al., 2024 ; Zhang et al., 2024b ) . SLIP ( Mu et al., 2022 ) introduces a multitask learning framework that integrates self-supervised learning with CLIP pretraining. FILIP ( Yao et al., 2021 ) enhances the fine-grained alignment between image patches and textual words by refining the contrastive loss, while maintaining the capability for offline pre-computation of image and text representations during inference, ensuring efficiency in both large-scale training and inference phases. ALIP ( Yang et al., 2023 ) proposes the use of synthetic captions and adaptive contrastive loss to mitigate the influence of noisy data and improve the efficacy of pre-training data utilization. SigLIP ( Zhai et al., 2023 ) utilizes a sigmoid loss, which supports scaling up batch sizes and also performs effectively at smaller batch sizes. However, global contrastive learning inadequately leverages compositional structures within image-text pairs, thereby limiting CLIP’s capability to capture nuanced compositional information in multimodal data ( Yuksekgonul et al., 2022 ) . As shown in Figure 1c , CLIP struggles to discern the relationship between “athlete” and “basketball”. Recent studies ( Yuksekgonul et al., 2022 ; Huang et al., 2024a ; Zhang et al., 2024a ; Doveh et al., 2023a ; Doveh et al., 2023b ) have sought to improve the compositional understanding of CLIP. NegCLIP ( Yuksekgonul et al., 2022 ) introduces the Attribution, Relation, and Order (ARO) benchmark and, for the first time, proposes a fine-tuning framework that integrates hard negative samples to enhance CLIP’s compositional understanding. Structure-CLIP ( Huang et al., 2024a ) integrates Scene Graph Knowledge (SGK) to augment multimodal structured representations. Hard-positives ( Kamath et al., 2024 ) incorporates hard positive samples during fine-tuning to strengthen the model’s capacity to capture subtle but semantically related variations among similar instances. CE-CLIP ( Zhang et al., 2024a ) introduces a simple yet effective fine-tuning framework with two fine-grained alignment losses to enhance compositional understanding, achieving state-of-the-art performance on multiple vision-language compositional reasoning benchmarks. However, our analysis reveals that existing methods improve CLIP’s compositional understanding at the cost of general capabilities, exhibiting significant catastrophic forgetting of pre-trained knowledge . To address the limitation of previous works, we innovatively introduce a De coupled G lobal- L ocal A lignment ( DeGLA ) framework, which aims to improve compositional understanding while substantially mitigating losses in general capabilities. To optimize the retention of the inherent capabilities of the model, we incorporate a self-distillation mechanism within the global alignment process, aligning the learnable image-text encoder with a teacher model derived from an exponential moving average. To improve compositional understanding, we first leverage the in-context learning capability of Large Language Models (LLMs) to construct about 2M high-quality negative captions across five types. Subsequently, we propose the Image-Grounded Contrast (ICC) and Text-Grounded Contrast (TCC) to improve fine-grained understanding. We conduct extensive experiments and demonstrate that our method achieves new state-of-the-art performance in both compositional and general tasks. The main contributions are summarized as follows: • We observe that previous methods, while enhancing CLIP’s compositional understanding, often compromise its general understanding capabilities . • We propose a simple yet effective negative caption generation pipeline that leverages the context learning capability of Large Language Models (LLMs) to generate high-quality negative captions. • We introduce the DeGLA framework , which employs a self-distillation mechanism within the global alignment to maintain the model’s inherent general comprehension capabilities. Additionally, it combines Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC) loss to improve vision-language compositional understanding. • We conduct extensive experiments and demonstrate that DeGLA achieves new state-of-the-art performance in both compositional and general tasks. 2. Related work 2.1. Vision-Language Contrastive Learning Vision-language representation learning aims to develop robust representations by pretraining models on extensive image-text pair datasets. A key development in this field, CLIP ( Radford et al., 2021b ) , pretrained on 400 million web-collected image-text pairs using contrastive learning, demonstrates strong zero-shot generalization across a wide range of vision tasks. Recent advancements include several methodologies that build on CLIP’s framework. UniCLIP ( Lee et al., 2022 ) improves data