Time-Contrastive Pretraining for In-Context Image and Video Segmentation
Assefa Wahd, Jacob Jaremko, Abhilash Hareendranathan · 2025
Read Time-Contrastive Pretraining for In-Context Image and Video Segmentation by Assefa Wahd, Jacob Jaremko, Abhilash Hareendranathan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17837.txt.
Introduction
Assefa Wahd, Jacob Jaremko, and Abhilash Hareendranathan present, in a 2025 paper, a pretraining trick that lets image models learn new segmentation tasks from a few examples. In-context learning promises quick adaptation, but the usual grid layout of examples limits how many can be shown and shrinks their detail.
Their method treats the task as video segmentation: examples and query become frames of one clip. A retriever, trained by contrasting nearby and distant video frames, picks the best example set for each query. On a public medical imaging test, the method beat baselines by wide margins on both still images and video, with overlap scores above ninety percent.
Catalog record
| CREATOR | Assefa Wahd, Jacob Jaremko, Abhilash Hareendranathan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Time-Contrastive Pretraining for In-Context Image and Video Segmentation Assefa Wahd, Jacob Jaremko, Abhilash Hareendranathan Abstract In-context learning (ICL) enables generalization to new tasks with minimal labeled data. However, mainstream ICL approaches rely on a gridding strategy, which lacks the flexibility required for vision applications. We introduce Temporal , a time-contrastive self-supervised objective that pretrains a prompt retriever for visual ICL, and formulate ICL as a video object segmentation (VOS) task. Temporal addresses key limitations of grid-based methods that restrict the number and resolution of context images. By reframing ICL as a VOS problem, our approach supports a variable number of context images while preserving their full resolution. To address the challenge of selecting optimal context sets for queries, we pretrain a prompt retriever on videos via self-supervised learning, where adjacent frames serve as positives and distant frames as negatives. For image segmentation, the prompt retriever selects relevant sequences that, when combined with the query, form coherent videos for VOS processing. For video segmentation, it identifies keyframes, predicts their masks using our ICL pipeline, and propagates them throughout the sequence. When evaluated on MICCAI FLARE 2022, our method achieves substantial improvements over baselines: 90.95% Dice score for image segmentation (10.64% improvement) and 92.45% Dice for video segmentation (14.88% improvement). 1 Introduction Figure 1: Model architecture. (a) Visual in-context learning via VOS: The retriever f_{\theta} selects the optimal context set P for a query image x_{q} . The query image x_{q} is appended to the context to create a video sequence, and the VOS model predicts the segmentation mask for x_{q} using this sequence. (b) Video segmentation with ICL: 1) Select keyframes (Section 2.3 ), 2) Predict masks for keyframes using step (a), 3) Propagate masks across the video. Medical imaging generates vast quantities of unlabeled data, yet acquiring dense annotations remains costly. This challenge has motivated the development of approaches requiring fewer labels, such as self-supervised learning [ 2 , 5 , 13 , 9 , 10 , 1 ] , semi-supervised learning [ 18 ] , and transfer learning [ 21 ] . Recently, foundation models have introduced new paradigms such as visual prompting and in-context learning (ICL) [ 22 , 7 , 8 , 3 , 23 ] , enabling adaptation to new tasks with minimal or no fine-tuning. In-context learning (ICL)—where models perform new tasks by conditioning on input-output examples without parameter updates—has been remarkably successful in natural language processing [ 4 ] . It offers a compelling vision for medical imaging: a single model could generalize across diverse segmentation tasks by "reasoning" over relevant annotated examples. However, adapting ICL to computer vision presents unique challenges. We highlight two critical issues: (1) selecting the most relevant context images, and (2) effectively modeling interactions between the context set and query images. An ideal visual ICL model must support a variable number of context images, and presever their full resolution. In textual tasks, input-output pairs can be trivially concatenated to form prompts; for visual ICL, however, it is less clear how to handle a variable number of images. Early attempts at visual ICL rely on a gridding strategy, in which input-output image pairs (i.e., the in-context examples) are arranged in a grid-like format alongside a query image [ 3 , 24 , 23 , 19 ] . However, this strategy limits the number of context images and necessitates downsampling, reducing image detail—a critical limitation for medical applications where fine details matter. More recent works have focused on context selection methods. Zhang et al. [ 24 ] found that selecting relevant context examples significantly impacts performance and proposed both unsupervised (CLIP-based [ 14 ] ) and supervised retrieval techniques. Subsequent innovations included padding images with learnable vectors [ 23 ] and combining channel and spatial features when computing similarity [ 17 ] . A promising alternative emerged when Foster et al. [ 8 ] framed ICL as a video object segmentation (VOS) task. In this approach, context images are treated as frames in a synthetic video, with the VOS model [ 6 ] segmenting the final frame (the query image). This allows handling full-resolution images and variable-sized context sets. However, they relied on pre-trained CLIP embeddings for context retrieval, which may not be optimal for domain-specific medical imaging tasks. Our work, named Temporal , extends this VOS-based framework with two key innovations: (1) a self-supervised time-contrastive approach to context retrieval and (2) a unified framework for both image and video segmentation. Our multipositive time-contrastive pretraining builds on Time-Contrastive Networks (TCN) [ 16 ] , enhanced with multipositive cross-entropy loss instead of traditional triplet loss. Unlike supervised retrieval methods [ 24 ] , we leverage temporal proximity in videos as a natural supervisory signal—temporally adjacent frames likely contain semantically similar information, creating positive pairs without manual labels. We provide a comprehensive analysis of context selection strategies, examining size, order, and diversity considerations, and introduce a diversity-aware retrieval mechanism using our time-contrastive network. For video data, we propose an efficient keyframe-based retrieval and propagation approach that significantly outperforms existing methods. To the best of our knowledge, Temporal is the first approach to combine time-contrastive networks with multipositive cross-entropy loss for self-supervised context retrieval from videos. While this paper focuses on using this technique as a prompt retriever for visual in-context learning, future work will explore broader applications of multipositive TCNs as generic SSL networks. Our method achieves up to 92.45% Dice on video benchmarks—a 14.88% improvement over semi-supervised VOS baselines. Figure 1 illustrates our approach for both image and video segmentation. You can find the code at github.com/aswahd/Temporal . Figure 2: Multi-positive sampling strategy . For each anchor image in a clip, we sample M temporally adjacent frames as positives and all other frames in the clip as negatives. Shaded regions are frames that were not sampled in a minibatch. 2 Method Given a video sequence V=(x_{1},x_{2},\dots,x_{T}) where x_{t} represents a frame at time t , a VOS model \mathcal{V}_{\phi} generates segmentation masks \hat{y}_{t} by utilizing both current and historical frame information: \hat{y}_{t}=\mathcal{V}_{\phi}(x_{1},y_{1},x_{2},y_{2},\dots,x_{t-1},y_{t-1},x_{t}), (1) where \hat{y}_{t} is the predicted segmentation for frame t . Our approach builds on SAM2 [ 15 ] as the foundation VOS model, which incorporates: 1) ViT image encoder for embeddings, 2) Memory Encoder/Bank to store/update past predictions, 3) Memory Attention for historical context fusion, and 4) Promptable Mask Decoder integrating embeddings, prompts, and memory for segmentation. We refer readers to [ 15 ] for a detailed description of SAM2. 2.1 Visual In-Context Learning In-context learning (ICL) infers an output for a query input by conditioning on examples. Given a dataset \mathcal{D}=\{(x_{n},y_{n})\}_{n=1}^{N} of input-output pairs and a query x_{q} , we predict \hat{y}_{q} using a model \mathcal{V}_{\phi} conditioned on a context P=\{(x_{c_{1}},y_{c_{1}}),\dots,(x_{c_{K}},y_{c_{K}})\}\subseteq\mathcal{D}. To generalize beyond grid layouts, we treat P and x_{q} as frames in a synthetic video , where each pair (x_{c_{k}},y_{c_{k}}) is a labeled frame and x_{q} is the unlabeled final frame. A VOS model then segments x_{q} : \mathcal{V}_{\phi}(P,x_{q})=\mathcal{V}_{\phi}\bigl(x_{c_{1}},y_{c_{1}},x_{c_{2}},y_{c_{2}},\dots,x_{c_{K}},y_{c_{K}},\,x_{q}\bigr), (2) where \mathcal{V}_{\phi} is a pretrained foundation model. A prompt retriever f_{\theta} selects the optimal context as outlined in the next section. 2.2 Multipositive Time-Contrastive Pretraining Our objective is to sample multiple positive examples along with a substantial number of negatives for each anchor frame in a