Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

Shizhan Gong, Yankai Jiang, Qi Dou, Farzan Farnia · 2025

Read Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models by Shizhan Gong, Yankai Jiang, Qi Dou, Farzan Farnia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02557.txt.

Introduction

Shizhan Gong and three co-authors present a method from 2025 to sharpen the image sense of a popular vision-and-text model called CLIP. CLIP links pictures and words well, but it often misses fine detail in a scene. A second model, DINOv2, sees fine detail much better but does not work with text.

The new method trains only the image side of CLIP so its outputs line up with those of DINOv2, using a kernel-based goal that suits step-by-step training. Text handling stays frozen and still matches. After training, the model did better at naming objects with no prior examples, at judging where things sit, and at finding objects in a scene. Larger chat models that used the tuned encoder also improved.

Catalog record

CREATORShizhan Gong, Yankai Jiang, Qi Dou, Farzan Farnia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models Shizhan Gong, Yankai Jiang, Qi Dou, Farzan Farnia Abstract Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essential components of many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo. However, numerous studies have identified CLIP’s limited fine-grained perception as a critical drawback, leading to substantial failures in downstream MLLMs. In contrast, vision-centric foundation models like DINOv2 demonstrate remarkable capabilities in capturing fine details from images. In this work, we propose a novel kernel-based method to align CLIP’s visual representation with that of DINOv2, ensuring that the resulting embeddings maintain compatibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization. By integrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance. The code and models are available at https://github.com/peterant330/KUEA . 1 Introduction Vision-language Models (VLMs) have made significant strides and transformed the field of computer vision. A notable example is CLIP ( Radford et al. 2021 ) and its variants ( Zhai et al. 2023 ; Sun et al. 2023 ) , which are trained on extensive datasets of paired text and images to link images with their corresponding textual descriptions. These models demonstrate exceptional generalizability and zero-shot performance on various downstream tasks, including classification ( Saha et al. 2024 ) , segmentation ( Yu et al. 2023 ) , and object detection ( Vidit et al. 2023 ) . Beyond functioning as a standalone tool, CLIP’s vision encoder has been incorporated into several Multi-modal Large Language Models (MLLMs), such as LLaVA ( Liu et al. 2024 ) , OpenFlamingo ( Awadalla et al. 2023 ) , BLIP-2 ( Li et al. 2023a ) , and Qwen-VL ( Bai et al. 2023 ) , serving as an integral component for visual feature extraction. Figure 1 : Main claim of this work. We propose a kernel-based alignment framework, which is able to enhance the visual representation of CLIP via image-only fine-tuning. Moreover, the improvement can be transferred to downstream multi-modal large language models. However, because CLIP is trained with global supervision from image captions, it struggles to learn finer pixel-level details such as color ( Thrush et al. 2022 ) and spatial relationship ( Liu et al. 2023 ; Kamath et al. 2023 ) . This limitation affects CLIP’s performance on vision-focused tasks and can also impair the fine-grained perception capabilities of downstream MLLMs. Numerous studies have highlighted these issues ( Yuksekgonul et al. 2023 ; Guo et al. 2024 ) . For instance, Jiang et al. 2023 examined feature visualization from deeper layers and found that it emphasizes global image properties while neglecting intricate details. Similarly, Tong et al. 2024 reported that current MLLMs struggle with simple visual pattern questions, such as counting, color identification, and viewpoint recognition. Vision-only self-supervised learning protocols, such as DINO ( Caron et al. 2021 ) and MAE ( He et al. 2022 ) , produce vision-centric representations that are highly effective for visual grounding. Researchers are exploring these models to address the inherent limitations of CLIP. Some studies have combined multiple models as vision encoders for MLLMs ( Jiang et al. 2023 ; Tong et al. 2024 ; Shi et al. 2024 ; Shen et al. 2024 ) . However, this approach introduces considerable computational overhead. Other methods focus on using vision-only self-supervised learning to fine-tune the CLIP encoder ( Wu et al. 2023 ; Covert et al. 2025 ) , which can enhance its localization abilities. Nevertheless, these fine-tuning techniques risk disrupting the alignment between image and text representations, potentially undermining CLIP’s zero-shot performance. Additionally, there are efforts to integrate region-based task loss ( Zhong et al. 2022 ; Wan et al. 2024 ) or distillation loss from vision models ( Salehi et al. 2023 ) during CLIP’s training, which requires re-training the models and incurs high computational costs. Most critically, all these methods produce visual embeddings that differ significantly from the original CLIP embeddings, leading to incompatibility with downstream models trained on the original embeddings. Consequently, all downstream models (e.g., MLLMs) would need to be re-tuned from scratch to align visual-text embeddings. The cost makes these solutions impractical in real applications. In this paper, we present an innovative approach to address the limitations of CLIP embeddings by aligning them with embeddings from vision-centric models. Our method involves fine-tuning CLIP’s vision branch to align with the representations of target models, such as DINOv2 ( Oquab et al. 2024 ) , while maintaining compatibility with CLIP’s text embeddings, which is totally frozen and untouched (see Fig. 1 ). Given the significant differences between the feature spaces of target models and CLIP, a direct alignment of the visual representations could disrupt the alignment with text representations. Instead, we propose to align the embeddings in the kernel space, which preserves the integrity of the original feature space while allowing for flexible adjustments of similarities among samples based on their visual details. As a result, this alignment could enhance the vision encoder’s ability to recognize fine-grained visual patterns. In addition, we design an optimization objective that can be handled by stochastic optimization, enabling our framework to scale effectively to real-world datasets with minimal computational hardware requirements. We examine the enhancements achieved through the proposed alignment in both vision-centric tasks and visual question answering (VQA). Experiments conducted on multiple CLIP-benchmark ( LAION-AI 2022 ) and probing bench ( Covert et al. 2025 ) reveal that the CLIP after alignment demonstrates improved accuracy in zero-shot classification and dense prediction tasks, without requiring fine-tuning of the text encoder. Subsequently, we integrate the aligned CLIP vision encoder into two pre-trained MLLMs, LLaVA ( Liu et al. 2024 ) and OpenFlamingo ( Awadalla et al. 2023 ) , evaluating their performance across several standard VQA benchmarks. This also results in significant enhancements over the original CLIP, even without fine-tuning the large language model (LLM) component. Our main contributions can be summarized as follows. • We introduce a kernel-based alignment method that effectively aligns two sets of embeddings while preserving the integrity of the original representation space. • We implement the alignment using CLIP and DINOv2. With minimal image-only fine-tuning, the aligned CLIP visual encoder shows substantial improvements while preserving compatibility with the text encoder and ensuring zero-shot generalizability. • Evaluation across various vision-language benchmarks shows that the enhancements in visual representations can be effectively inherited to downstream MLLMs. 2 Related Work Kernel Methods in Representation Learning. Kernel methods ( Hofmann et al. 2008 ) are algorithms that use the kernel trick, which allows linear algorithms to work in high-dimensional feature spaces without explicit data transformation. It is widely used in deep representation learning. For example, He & Ozay 2022 utilized kernel-based metrics for evaluating the similarity between two embeddings. Several work ( Allen-Zhu & Li 2020 ; He & Ozay 2022 ; Zhou et al. 2024 ) leveraged kernel functions for knowledge distillation. Dehdashtian et al. 2024 proposed to de-bias CLIP’s image and text representations in reproducing

Read the complete text