ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder

Xiaoxing Hu, Kaicheng Yang, Ziyang Gong, Qi Ming, Zonghao Guo, Xiang An, Ziyong Feng, Junchi Yan, Xue Yang · 2025

Read ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder by Xiaoxing Hu, Kaicheng Yang, Ziyang Gong, Qi Ming, Zonghao Guo, Xiang An, Ziyong Feng, Junchi Yan, Xue Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18795.txt.

Introduction

Xiaoxing Hu and his eight co-authors fix a known weakness of CLIP. CLIP links images and text. Its text encoder takes at most seventy-seven tokens. Long descriptions get cut off. Fine detail is lost. Languages beyond English are poorly served. A large language model embedder could replace that encoder. In principle, the swap helps a lot. But the two spaces were trained apart. Forcing them together can damage learned alignment.

Their framework, ProCLIP, aligns the pieces step by step. The pace follows a curriculum. First, CLIP's text knowledge is distilled into the new embedder. That builds a bridge to the image encoder. Next, contrastive training completes the alignment. A self-distillation guardrail protects the old knowledge. The goal is a model that reads long, multilingual text. It should keep CLIP's original strengths as well.

Catalog record

CREATORXiaoxing Hu, Kaicheng Yang, Ziyang Gong, Qi Ming, Zonghao Guo, Xiang An, Ziyong Feng, Junchi Yan, Xue Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18795

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder Xiaoxing Hu, Kaicheng Yang, Ziyang Gong, Qi Ming, Zonghao Guo, Xiang An, Ziyong Feng, Junchi Yan, Xue Yang Abstract The original CLIP text encoder is limited by a maximum input length of 77 tokens, which hampers its ability to effectively process long texts and perform fine-grained semantic understanding. In addition, the CLIP text encoder lacks support for multilingual inputs. All these limitations significantly restrict its applicability across a broader range of tasks. Recent studies have attempted to replace the CLIP text encoder with an LLM-based embedder to enhance its ability in processing long texts, multilingual understanding, and fine-grained semantic comprehension. However, because the representation spaces of LLMs and the vision-language space of CLIP are pretrained independently without alignment priors, direct alignment using contrastive learning can disrupt the intrinsic vision-language alignment in the CLIP image encoder, leading to an underutilization of the knowledge acquired during pre-training. To address this challenge, we propose ProCLIP , a curriculum learning-based progressive vision-language alignment framework to effectively align the CLIP image encoder with an LLM-based embedder. Specifically, ProCLIP first distills knowledge from CLIP’s text encoder into the LLM-based embedder to leverage CLIP’s rich pretrained knowledge while establishing initial alignment between the LLM embedder and CLIP image encoder. Subsequently, ProCLIP further aligns the CLIP image encoder with the LLM-based embedder through image-text contrastive tuning, employing self-distillation regularization to avoid overfitting. To achieve a more effective alignment, instance semantic alignment loss and embedding structure alignment loss are employed during representation inheritance and contrastive tuning. Extensive experiments show ProCLIP achieves 6.8% to 13.5% improvement on zero-shot classification and presents excellent performance on cross-modal retrieval, multilingual cross-modal retrieval, and fine-grained understanding tasks, demonstrating the effectiveness and robustness of ProCLIP. The Code is available at https://github.com/VisionXLab/ProCLIP . 1 introduction CLIP demonstrates remarkable zero-shot recognition capabilities by learning joint vision-language representations through contrastive learning on large-scale image–text pairs ( Radford et al. 2021 ) . Serving as a bridge between vision and language, it is widely adopted in multiple downstream tasks such as image–text retrieval ( Yang et al. 2023 ; Zheng et al. 2025 ) , text-to-image generation ( Wang et al. 2022 ) , and open-vocabulary object detection ( Wu et al. 2023b ) . However, the original CLIP model relies on English text captions with a maximum length of 77 tokens as its supervisory signal ( Zhang et al. 2024 ) . This design limits its capacity to process long-form text and restricts input to English-only ( Tschannen et al. 2025 ) . Additionally, due to the absence of supervision for fine-grained textual semantics, this limitation further impedes its semantic understanding capability ( Hu et al. 2025 ; Gu et al. 2025a ; Gu et al. 2025b ) . To overcome these limitations, methods such as Long-CLIP ( Zhang et al. 2024 ) interpolate positional embeddings and fine-tune on long text–image pairs to extend the input length. While effective for long-text understanding, such approaches still fall short in enhancing CLIP’s fine-grained semantic understanding and multilingual capabilities. Recently, LLM exhibits remarkable proficiency in natural language processing, and it has pivoted towards harnessing decoder-only architectures for effective representation learning ( BehnamGhader et al. 2024 ; Lee et al. 2024 ) . Following this trend, methods such as FLAME ( Cao et al. 2025 ) and LLM2CLIP ( Huang et al. 2024 ) propose to replace CLIP’s original text encoder with LLM-based embedders. By leveraging the rich open-world knowledge inherent in LLMs, these approaches aim to enhance CLIP’s representational capacity—particularly in processing longer and more complex image captions. However, these methods align the CLIP image encoder directly with the LLM-based text embedder through contrastive learning, while neglecting the rich pretrained knowledge within CLIP. This “ from-scratch alignment ” compels both encoders to learn a new representation space from scratch, disregarding the original CLIP alignment knowledge. Such an approach increases the risk of overfitting, particularly when training data is scarce, thereby compromising model generalization. This observation leads to a critical research question: How can we systematically leverage CLIP’s pretrained knowledge to achieve efficient cross-modal alignment with an LLM-based embedder while preserving generalization capability? In this paper, we propose ProCLIP , a simple yet effective progressive vision-language alignment framework enhancing the CLIP. ProCLIP leverages curriculum learning to first guide the LLM-based embedder (only MLP trainable) to adapt to the CLIP text encoder’s representation space, and then uses contrastive learning to further learn joint image-text representations. Specifically, ProCLIP first distills knowledge from the original CLIP text encoder into the LLM-based embedder, establishing an initial alignment between the CLIP image encoder and LLM-based embedder. Subsequently, we conduct contrastive learning on image–text pairs to further improve this alignment. Since the LLM-based embedder is already partially aligned with the CLIP image encoder during the prior stage, the contrastive optimization process becomes more stable and preserves generalization more effectively. To further mitigate overfitting, we impose a self-distillation constraint on the CLIP image encoder throughout this stage, which stabilizes training and improves generalization. To prove the effectiveness of ProCLIP, we evaluate it on multiple tasks across diverse data scales and model sizes. Extensive experiment results demonstrate that ProCLIP achieves consistently significant improvements. The main contributions of this paper are summarized as follows: • We highlight the limitation of previous works : previous methods fail to fully exploit the pre-trained knowledge in CLIP, and their reliance on simplistic contrastive learning for cross-modal alignment significantly compromises CLIP’s inherent generalization capabilities. • We propose ProCLIP a simple but effective Progressive vision-language alignment framework to enhance CLIP . ProCLIP initially distills the pretrained knowledge into the LLM-based embedder. After that, ProCLIP utilizes contrastive fine-tuning constrained by self-distillation to further enhance cross-modal alignment while preserving the model’s inherent knoweledge. • We conduct extensive experiments on multiple tasks across diverse scales of data and model . Compared to the baseline, ProCLIP achieves 6.8% to 13.5% improvement on zero-shot classification and performs strongly on other tasks, including short-text cross-modal retrieval, long-text cross-modal retrieval, multilingual cross-modal retrieval, and fine-grained understanding. Figure 1: Illustration of research gap. Previous work directly aligns the LLM-based embedder with the CLIP image encoder, disregarding the valuable knowledge embedded in the pre-trained CLIP model. In contrast, ProCLIP first transfers knowledge from CLIP’s text encoder to the LLM embedder via distillation, establishing an initial alignment. It then refines the alignment between the CLIP image encoder and the LLM-based embedder through image-text contrastive learning with self-distillation regularization. 2 related work Vision-Language Contrastive Learning. Vision-language contrastive learning aims to learn robust multimodal representations by pretraining on large-scale image-text pairs. A seminal work in this area, CLIP ( Radford et al. 2021 ) aligns visual and linguistic representations through contrastive learning, bridging both modalities in a shared semantic space. As a bridge between vision and language, CLIP has been widely applied in multimodal learning. It enables a variety of natural language-guided open-vocabulary recognition tasks, including image classification ( Zhou et al. 2022b ; Zhou et al. 2022a ; Kim et al. 2024 ) , open-vocabulary semantic segmentation ( Ding et al. 2022 ; Li et al. 2022 ; Ghiasi et al. 2022 ; Xu et al. 2022 ; Cho et al. 2024 ; Lan et al. 2024 ) , and

Read the complete text