TIPSv2 : Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Bingyi Cao Koert Chen Kevis-Kokitsi Maninis Kaifeng Chen Arjun Karpur, Ye Xia Sahil Dua Tanmaya Dabral Guangxing Han Bohyung Han Joshua Ainslie, Alex Bewley Mithun Jacob René Wagner Washington Ramos Krzysztof Choromanski, Mojtaba Seyedhosseini Howard Zhou André Araujo, Google DeepMind · 2026

Read TIPSv2 : Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment by Bingyi Cao Koert Chen Kevis-Kokitsi Maninis Kaifeng Chen Arjun Karpur, Ye Xia Sahil Dua Tanmaya Dabral Guangxing Han Bohyung Han Joshua Ainslie, Alex Bewley Mithun Jacob René Wagner Washington Ramos Krzysztof Choromanski, Mojtaba Seyedhosseini Howard Zhou André Araujo, Google DeepMind (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12012.txt.

Introduction

Cao and eighteen colleagues advance vision-language pretraining with TIPSv2, a family of image-text encoders tackling patch-text alignment. Patch-level distillation strongly boosts dense patch-text alignment, surprisingly pushing the student past its teacher. That insight drives iBOT++, where unmasked tokens also feed the masked image loss, dramatically improving alignment. Tweaks to the moving-average setup and caption sampling across granularities complete the recipe. Across nine tasks and twenty datasets, TIPSv2 performs on par with or better than recent vision encoders.

Catalog record

CREATORBingyi Cao Koert Chen Kevis-Kokitsi Maninis Kaifeng Chen Arjun Karpur, Ye Xia Sahil Dua Tanmaya Dabral Guangxing Han Bohyung Han Joshua Ainslie, Alex Bewley Mithun Jacob René Wagner Washington Ramos Krzysztof Choromanski, Mojtaba Seyedhosseini Howard Zhou André Araujo, Google DeepMind
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56105
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TIPSv2 : Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment Bingyi Cao Koert Chen Kevis-Kokitsi Maninis Kaifeng Chen Arjun Karpur, Ye Xia Sahil Dua Tanmaya Dabral Guangxing Han Bohyung Han Joshua Ainslie, Alex Bewley Mithun Jacob René Wagner Washington Ramos Krzysztof Choromanski, Mojtaba Seyedhosseini Howard Zhou André Araujo, Google DeepMind Abstract Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment – surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/ . 1 Introduction Figure 1 : TIPSv2’s improvement to the masked image modeling pretraining strategy. As part of our complete TIPSv2 method, we introduce iBOT++ (bottom), a simple modification to the well-known iBOT Zhou et al. 2022b self-supervised objective (top-left), where visible tokens also contribute directly to the loss. This enhancement dramatically improves patch-text alignment, as demonstrated by zero-shot image segmentation results (top-right). Large-scale self-supervised and vision-language pretraining methods have fundamentally reshaped visual representation learning, establishing new performance standards across a broad spectrum of tasks. Leading approaches generally fall into two categories: weakly-supervised contrastive or sigmoid methods, exemplified by CLIP ( Radford et al. 2021 ) , SigLIP ( Zhai et al. 2023 ) and Perception Encoder (PE) ( Bolya et al. 2025 ) , which provide robust image-text alignment and zero-shot capabilities; and self-supervised learning (SSL) methods, such as DINO ( Siméoni et al. 2025 ; Caron et al. 2021 ; Oquab et al. 2024 ) and iBOT ( Zhou et al. 2022b ) , which excel at spatial understanding for dense downstream tasks (e.g., depth, segmentation). Achieving a unified representation that excels simultaneously at global (image-level) and dense (patch-level) understanding remains a significant challenge. State-of-the-art (SOTA) models often optimize one at the expense of the other: for example, DINOv2 ( Oquab et al. 2024 ) excels at dense vision tasks but lacks inherent text alignment for vision-language capabilities, whereas PE-core ( Bolya et al. 2025 ) provides robust image-text alignment but underperforms on dense tasks. Recent unified approaches, such as TIPS ( Maninis et al. 2025 ) and SigLIP2 ( Tschannen et al. 2025 ) , have made significant strides in bridging this divide. However, we find that even these modern models struggle to maintain precise pixel- or patch-level alignment with text—a capability essential for advanced tasks like open-vocabulary segmentation. Self-supervised works such as DINO.txt ( Jose et al. 2025 ) and DINOv3 ( Siméoni et al. 2025 ) attempt to address this by training supplementary text encoders, but still exhibit limited performance on multimodal tasks. Overall, grounded alignment with text remains a challenging task. This is corroborated by PE ( Bolya et al. 2025 ) , which finds that final transformer layers often function as global contrastive “decoders”, rather than preserving the local semantics. In this work, we first uncover a surprising trend in SOTA vision encoders ( Maninis et al. 2025 ; Tschannen et al. 2025 ) : the largest flagship models often underperform their smaller counterparts in patch-text alignment. Investigating this issue with a recent spatially-aware multimodal encoder ( Maninis et al. 2025 ) , we find that distillation significantly improves grounding alignment by applying effective supervision across all patch tokens. This insight not only explains the performance gap, but also helps us unlock broader capabilities through a novel pretraining recipe. We introduce TIPSv2 , the second version of the TIPS Maninis et al. 2025 model family ( T ext- I mage P retraining with S patial awareness), a novel methodology designed to bridge this gap directly during pretraining. TIPSv2 incorporates a new objective, iBOT++, which translates our distillation findings into a self-supervised pretraining objective that enforces representation consistency on visible tokens, as per Fig. 1 . We further enhance this with multi-granularity text augmentation (leveraging captions from PaliGemma ( Beyer et al. 2024 ) and Gemini ( Gemini Team, Google 2024 ) ) and a resource-efficient Exponential Moving Average (EMA) strategy. While standard SSL often requires full-model EMA, the presence of text supervision allows us to use a ‘head-only EMA’. This updates only the projection layers, drastically reducing memory requirements at training time. TIPSv2 provides an efficient training framework yielding a highly effective image-text encoder across standard benchmarks. It sets new SOTA results in zero-shot semantic segmentation, while providing strong performance on a wide range of downstream multimodal tasks. In summary, our primary contributions are: • A novel finding that a distillation procedure tuned to strengthen spatial awareness unlocks strong gains in patch-text alignment. Our detailed study pinpoints masking removal and initialization as the key factors driving this phenomenon. • iBOT++, a novel self-supervised masked objective that enhances pretraining to enforce representation consistency and significantly improves patch-text alignment – see Fig. 1 . • An efficient and effective training recipe leveraging multi-granularity text augmentation and a novel head-only EMA mechanism, overall achieving strong performance with significantly reduced training cost, validated comprehensively on 9 tasks spanning 20 datasets. 2 Related Work There are currently three dominant paradigms for large-scale pretraining of versatile image representations: self-supervised visual learning Chen et al. 2020 ; Assran et al. 2023 ; He et al. 2022 , image-text contrastive learning Radford et al. 2021 ; Jia et al. 2021 ; Cherti et al. 2023 or image captioning Tschannen et al. 2023 ; Wan et al. 2024 ; Fini et al. 2025 . In this work, we build on top of self-supervised and image-text contrastive techniques to construct visual representations that are natively text-aligned and spatially aware. We review relevant work in these areas and contextualize our advances with respect to them. Self-supervised learning methods enable visual pretraining without the use of any labels, by formulating pretext tasks that derive supervisory signals from the images themselves. Landmark approaches in this area, such as DINO Caron et al. 2021 and iBOT Zhou et al. 2022b , enforce representational invariances to image augmentations such as crops, flips, color distortions and masking. These methods adopt a student-teacher distillation setup, where the teacher model is constructed as an exponential moving average (EMA) of the student He et al. 2020 . DINOv2

Read the complete text