DIP: Unsupervised D ense I n-Context P ost-training of Visual Representations
Sophia Sirko-Galouchenko Spyros Gidaris, Antonin Vobecky Andrei Bursuc Nicolas Thome · 2025
Read DIP: Unsupervised D ense I n-Context P ost-training of Visual Representations by Sophia Sirko-Galouchenko Spyros Gidaris, Antonin Vobecky Andrei Bursuc Nicolas Thome (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18463.txt.
Introduction
Sophia Sirko-Galouchenko and colleagues proposed a cheap way in 2025 to make a pretrained vision model's dense, pixel-level features smarter about scenes. In-context scene understanding asks a model to segment new kinds of regions from a few examples shown at test time, and it lives or dies on the quality of those dense representations. Earlier improvements leaned on elaborate self-distillation machinery; this one instead rehearses the test itself.
Their method, DIP, post-trains the encoder on pseudo-tasks that mimic downstream in-context problems, generated automatically by pairing a pretrained diffusion model with the encoder's own current features, a trick borrowed from meta-learning. No labels are used and the whole run takes under nine hours on a single high-end graphics card. Across a wide range of real in-context segmentation tasks, the rehearsed encoder beat both its untouched starting point and prior methods.
Catalog record
| CREATOR | Sophia Sirko-Galouchenko Spyros Gidaris, Antonin Vobecky Andrei Bursuc Nicolas Thome |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18463
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DIP: Unsupervised D ense I n-Context P ost-training of Visual Representations Sophia Sirko-Galouchenko Spyros Gidaris, Antonin Vobecky Andrei Bursuc Nicolas Thome Abstract We introduce DIP , a novel unsupervised post-training method designed to enhance dense image representations in large-scale pretrained vision encoders for in-context scene understanding. Unlike prior approaches that rely on complex self-distillation architectures, our method trains the vision encoder using pseudo-tasks that explicitly simulate downstream in-context scenarios, inspired by meta-learning principles. To enable post-training on unlabeled data, we propose an automatic mechanism for generating in-context tasks that combines a pretrained diffusion model and the vision encoder itself. DIP is simple, unsupervised, and computationally efficient, requiring less than 9 hours on a single A100 GPU. By learning dense representations through pseudo in-context tasks, it achieves strong performance across a wide variety of downstream real-world in-context scene understanding tasks. It outperforms both the initial vision encoder and prior methods, offering a practical and effective solution for improving dense representations. Code available here 1 Introduction Our goal is to learn dense image representations from unlabeled data for effective in-context scene understanding. In-context learning, which enables models to adapt to tasks without updating parameters, has seen remarkable success in large language models (LLMs) [ 7 ] . By providing a few examples within the input prompt, LLMs generalize effectively to new tasks. Inspired by this, recent efforts aim to bring similar in-context learning capabilities to vision-only models [ 3 , 5 ] . A notable approach by Balazevic et al. [ 3 ] reformulates dense prediction tasks as nearest-neighbor retrieval problems using patch feature similarities, demonstrating strong performance, especially with limited data. However, while recent self-supervised Vision Transformers (ViTs) [ 3 , 56 ] show promise for in-context scene understanding, they still fall short of matching LLMs’ in-context learning success. To bridge this gap, it is crucial to learn dense features that establish strong semantic correspondences between patches in test and training images. A key insight from prior work [ 58 , 83 , 64 ] is that leveraging pretrained Vision Foundation Models (VFMs), such as DINOv2R [ 56 , 16 ] , and specializing them through unsupervised post-training is more computationally efficient and effective than training from scratch. This aligns with trends in LLMs, where large-scale pretraining is followed by cheaper post-training stages to specialize in specific skills of interest. We refer to the pretrained VFM undergoing post-training as the base model . Most existing methods, whether training from scratch or post-training, rely on self-distillation, where the teacher is an exponential moving average of the student weights. These frameworks use architectures and objectives designed to promote dense nearest-neighbor retrieval skills. For instance, state-of-the-art methods enforce consistency between objects across images and views [ 46 ] or between patch rankings from two random views of the same image [ 58 ] . However, self-distillation often suffers from poor stability [ 82 ] and hyperparameter sensitivity [ 2 ] . Moreover, these approaches introduce additional complexity, such as differentiable sorting [ 58 ] , RoI alignment units [ 58 , 83 , 29 ] , Sinkhorn-Knopp optimization [ 83 , 64 ] , or learnable patch pooling [ 3 , 46 ] . While improving performance, these components make the methods harder to interpret and tune, and less transparent in terms of their underlying mechanisms. In this work, we adopt a post-training approach due to it’s computational efficiency and effectiveness. Within this context, we address two key questions: (1) Are there simpler and more effective alternatives to existing self-distillation frameworks with complex architectures and objectives? , and (2) Beyond the base model, can we leverage other vision foundation models to achieve our goal? We propose a simpler post-training approach with an unsupervised learning objective explicitly designed for in-context scene understanding. Inspired by meta-learning, which learns general knowledge from diverse tasks to address unseen but related tasks, we automatically construct multiple in-context tasks from unlabeled data and “meta-train” the model on them. This enables the model to acquire transferable dense representations for efficiently solving new in-context tasks during downstream stages. Unlike prior methods, our approach eliminates complex training objectives and architectures, avoiding reliance on self-distillation frameworks. To achieve unsupervised post-training, we generate in-context pseudo-tasks that mimic the tasks encountered during downstream use. Each pseudo-task consists of support examples with segmentation masks and a query image to be segmented based on the support examples. The support set includes a positive example (sharing objects with the query) and “distractor” examples (likely unrelated). Our pseudo-tasks are analogous to episodes from meta-learning literature [ 75 ] with the difference that we do not use any manual annotations during the post-training stage. With each episode, the model is essentially rehearsing for the downstream in-context segmentation task. We leverage pretrained VFMs to construct these pseudo-tasks. Specifically, apart from the base model (i.e., DINOv2R) that we post-train, we also use an auxiliary pretrained generative model, namely Stable Diffusion (SD) [ 27 ] . SD provides high-fidelity, class-agnostic image segments in an unsupervised manner, while the base model retrieves candidate positive examples for each query image and assigns pseudo-labels to the extracted segments. Although SD was trained on weakly annotated (internet-scraped) image-caption pairs, we consider our post-training method as unsupervised, as neither it nor the used pretrained models rely on dense segmentation labels. The main contributions of this paper are as follows: • We introduce DIP , a novel unsupervised post-training method that uses retrieval-based in-context learning to improve dense image representations. Unlike prior work, DIP is based on meta-training principles and eliminates the need for complex self-distillation architectures. Furthermore, DIP is computationally efficient, requiring less than 9 hours on a single A100 GPU for post-training. • To enable post-training on unlabeled data, we propose an automatic mechanism for generating in-context tasks. In addition to the base model, this approach leverages the Stable Diffusion model (through the unsupervised, training-free DiffCut [ 15 ] technique) to produce high-quality pseudo-segmentations from unlabeled images. • Experiments show that the learned dense representations generalize effectively to a wide variety of downstream tasks. These include six semantic segmentation datasets and one monocular depth prediction dataset, even in low-shot scenarios. Furthermore, we extend our approach to other VFMs, such as CLIP [ 61 ] , and show that it consistently enhances their dense representations. • Compared to DINOv2R, DIP produces more semantically coherent neighbors, leading to more accurate retrieval-based scene understanding, particularly in low-shot settings (see ). Additionally, DIP outperforms recent state-of-the-art post-training methods in most retrieval-based tasks while being significantly simpler. 2 Related Work Self-supervised learning. Self-supervised approaches can learn rich representations from vast amounts of raw data via pretext tasks that provide supervision signals from the data itself. The model is subsequently fine-tuned on a downstream task of interest with few annotated examples. Self-supervision is commonly applied by hiding or transforming the input data and training the model to predict the missing parts or correct them. We distinguish several broad categories of self-supervised objectives for images: contrastive objectives for contrasting representations between visually similar and dissimilar views [ 54 , 11 , 30 ] , clustering-based objectives [ 1 , 8 , 23 ] , self-distillation objectives for predicting similar representations for different views of the