PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning
Yicheng Xiao, Yu Chen, Haoxuan Ma, Jiale Hong, Caorui Li, Lingxiang Wu, Haiyun Guo, Jinqiao Wang · 2025
Read PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning by Yicheng Xiao, Yu Chen, Haoxuan Ma, Jiale Hong, Caorui Li, Lingxiang Wu, Haiyun Guo, Jinqiao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04601.txt.
Introduction
Yicheng Xiao leads eight authors with PixCLIP, a method for fine-grained vision-language understanding. CLIP links images and text well at a broad level, but fine detail alignment lags. Most fixes add visual prompts that steer focus to chosen regions.
PixCLIP instead trains with pixel-level alignment at any chosen granularity. Long, detailed captions help, yet CLIP's text encoder has a strict token limit that blocks them. The authors work around that cap so detailed text can guide local image features. Sharper alignment should help tasks like grounding a phrase to the exact part of a picture.
Catalog record
| CREATOR | Yicheng Xiao, Yu Chen, Haoxuan Ma, Jiale Hong, Caorui Li, Lingxiang Wu, Haiyun Guo, Jinqiao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04601
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning Yicheng Xiao, Yu Chen, Haoxuan Ma, Jiale Hong, Caorui Li, Lingxiang Wu, Haiyun Guo, Jinqiao Wang Abstract While the Contrastive Language-Image Pretraining(CLIP) model has achieved remarkable success in a variety of downstream vison language understanding tasks, enhancing its capability for fine-grained image-text alignment remains an active research focus. To this end, most existing works adopt the strategy of explicitly increasing the granularity of visual information processing, e.g., incorporating visual prompts to guide the model focus on specific local regions within the image. Meanwhile, researches on Multimodal Large Language Models(MLLMs) have demonstrated that training with long and detailed textual descriptions can effectively improve the model’s fine-grained vision-language alignment. However, the inherent token length limitation of CLIP’s text encoder fundamentally limits CLIP to process more granular textual information embedded in long text sequences. To synergistically leverage the advantages of enhancing both visual and textual content processing granularity, we propose PixCLIP, a novel framework designed to concurrently accommodate visual prompt inputs and process lengthy textual descriptions. Specifically, we first establish an automated annotation pipeline capable of generating pixel-level localized, long-form textual descriptions for images. Utilizing this pipeline, we construct LongGRIT, a high-quality dataset comprising nearly 1.5 million samples. Secondly, we replace CLIP’s original text encoder with the LLM and propose a three-branch pixel-text alignment learning framework, facilitating fine-grained alignment between image regions and corresponding textual descriptions at arbitrary granularity. Experiments demonstrate that PixCLIP showcases breakthroughs in pixel-level interaction and handling long-form texts, achieving state-of-the-art performance on fine-grained pixel-level tasks, while retaining the capabilities of original CLIP in handling global-level vision-language tasks. Code — https://github.com/StuHude/PixCLIP/ Introduction Figure 1: As shown on the (a), prior methods fail on fine-grained local tasks due to inability to accept mask input, While existing promptable methods struggle to handle long texts due to token limitations, as shown in (b). Only our method succeeds in yielding robust embeddings for image and text at any granularity. Contrastive Language-Image Pretraining (CLIP) ( Radford et al. 2021 ) is a multimodal model that jointly trains image and text encoders via contrastive learning to align their features in a shared semantic space. Its features aim to capture all the semantic details within images, demonstrating strong representation capabilities and exceptional generalizability. This inherent versatility makes them suitable for a variety of downstream tasks, such as open-world recognition ( Chen et al. 2023 ; Zhou, Loy, and Dai 2022 ; Lan et al. 2024 ) , retrieval ( Huang et al. 2024 ; Saito et al. 2023 ; Zhang et al. 2024 ) and Multimodal Large Language Models (MLLMs) ( Liu et al. 2023 ; Yang et al. 2024b ) . Although CLIP captures the content of the entire image, focusing on regions of interest is crucial for achieving a more refined understanding. Traditional image processing techniques, such as cropping regions or masking irrelevant parts of images, tend to destroy and overlook contextual information. As a result, recent works have focused on enhancing the regional representation of CLIP, enabling it to accept visual prompts that guide attention to specific regions of interest. Visionllm v2 ( Wu et al. 2024a ) , ControlMLLM ( Wu et al. 2024b ) , and Ferret ( You et al. 2023 ) have introduced prompt encoders to enable multimodal models to receive visual prompt inputs, but these models cannot directly obtain visual representations to perform tasks such as retrieval. CLOC ( Chen et al. 2025a ) achieves alignment through a large number of bounding box-text pairs. However, its capability is limited to bounding boxes, not masks, and thus fails to capture the fine-grained, pixel-level details that masks can offer. Alpha CLIP ( Sun et al. 2024 ) has addressed these issues by adding an alpha channel and using mask-text pairs for contrastive learning. However, since the training focused primarily on short and concise captions, these methods tend to struggle with longer and more detailed textual descriptions, performing poorly on tasks that require fine-grained analysis. So, is it possible to build a model that overcomes the limits to accept visual prompts for local regions and complex long texts, achieving image-text alignment at arbitrary granularity? In this paper, we propose PixCLIP, a universal CLIP model for fine-grained understanding through any-granularity pixel–text alignment learning, as shown in Fig. 11 . We innovatively perform pixel-level semantic alignment between masks and complex long texts. After fine-tuning, the model can dynamically embed any local region of an image and align it with complex texts of arbitrary length. PixCLIP achieves a breakthrough in both the visual and textual domains, surpassing traditional granularity limits, and establishes itself as a more fine-grained foundational model. Specifically, existing mask-text datasets like GRiT ( Peng et al. 2023 ) and FERRET ( You et al. 2023 ) are limited by short text and a lack of large-scale, well-aligned training data. To address this, we first propose LongGRIT , an automatically generated dataset of 1.5 million image-local fine-grained expressions designed to boost model performance. We also utilized 2,000 masked regions from LongGRIT to benchmark existing models for fine-grained local mask-text retrieval in complex settings, uncovering significant opportunities for improvement and further research. Secondly, we propose a multi-grained alignment framework. (1)We use an LLM as the text encoder and then employ contrastive learning with mask-text pairs. This enables the model to directly obtain embeddings from images with mask and align them with fine-grained text features. (2)In parallel, we utilize Fine-grained Cropping Alignment. This branch extracts features solely from the region indicated by the mask. It aims to make these features more similar to the features from the first step , thereby enhancing the features to concentrate more strongly on the masked region. (3)Simultaneously, in the Local-Global Representation Enhancement branch, the mask covers the whole image, equivalent to extracting whole-image features and then mutually enhances the initial mask embedding by features from the corresponding region, jointly optimizing the representation capabilities for both the global and local. We conduct extensive experiments on RefCOCO ( Lin et al. 2015 ) , Ref-SAV ( Yuan et al. 2025 ) , Instance-COCO ( Lin et al. 2015 ) , ImageNet-S ( Gao et al. 2023 ) for region-level evaluation, as well as a series of image-text retrieval datasets such as DOCCI, Urban, and Flickr. The results demonstrate that our model successfully enables the processing of complex long texts while maintaining the ability to accept local visual cues, greatly improving the performance of fine-grained tasks. Our contribution is threefold: 1. We propose PixCLIP, a universal foundation model that extends the input to any local region and long texts. Extensive experiments demonstrate the superiority of this model in both region-level tasks and traditional tasks. 2. We construct and release the LongGRIT dataset, which contains 1.5 million detailed mask descriptions automatically generated and validated by SOTA MLLMs. 3. We propose a three-branch pixel-text alignment learning framework that significantly enhances the model’s performance through fine-grained cropping and local-global enhancement. Related Works Localized Image Representation To enable CLIP to disentangle regions from the whole image for more targeted processing and understanding, various methods have been explored in the field of segmentation. MaskCLIP ( Zhou, Loy, and Dai 2022 ) and ODISE ( Xu et