A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentation
1 st Qing Zhong 2 nd Peng-Tao Jiang * 3 rd Wen Wang University of Adelaide vivo Zhejiang University Adelaide, Australia Hangzhou, China Hangzhou, China 4 th Guodong Ding 5 th Lin Wu 6 th Kaiqi Huang NUS Swansea University University of South China Singapore Swansea, United Kingdom Hengyang, China · 2025
Read A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentation by 1 st Qing Zhong 2 nd Peng-Tao Jiang * 3 rd Wen Wang University of Adelaide vivo Zhejiang University Adelaide, Australia Hangzhou, China Hangzhou, China 4 th Guodong Ding 5 th Lin Wu 6 th Kaiqi Huang NUS Swansea University University of South China Singapore Swansea, United Kingdom Hengyang, China (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17672.txt.
Introduction
Qing Zhong and co-authors present a 2025 computing paper on teaching models to track objects in video. Most video segmentation tools first learn from still images, then adapt to video. That gap in time-based skill can hurt their work on hard clips with tangled objects.
Their fix is a pre-training plan built for video from the start. First, they build fake video clips from images while keeping each object steady across frames. Next, a module works at two scales to learn links across short and long spans of time. The plan fits many model designs without forcing a set shape. On standard test sets, the method beat prior top results, with a clear gain on the hard OVIS set.
Catalog record
| CREATOR | 1 st Qing Zhong 2 nd Peng-Tao Jiang * 3 rd Wen Wang University of Adelaide vivo Zhejiang University Adelaide, Australia Hangzhou, China Hangzhou, China 4 th Guodong Ding 5 th Lin Wu 6 th Kaiqi Huang NUS Swansea University University of South China Singapore Swansea, United Kingdom Hengyang, China |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentation 1 st Qing Zhong 2 nd Peng-Tao Jiang * 3 rd Wen Wang University of Adelaide vivo Zhejiang University Adelaide, Australia Hangzhou, China Hangzhou, China 4 th Guodong Ding 5 th Lin Wu 6 th Kaiqi Huang NUS Swansea University University of South China Singapore Swansea, United Kingdom Hengyang, China Abstract Contemporary Video Instance Segmentation (VIS) methods typically adhere to a pre-train then fine-tune regime, where a segmentation model trained on images is fine-tuned on videos. However, the lack of temporal knowledge in the pre-trained model introduces a domain gap which may adversely affect the VIS performance. To effectively bridge this gap, we present a novel “ video pre-training ” approach to enhance VIS models, especially for videos with intricate instance relationships. Our crucial innovation focuses on reducing disparities between the pre-training and fine-tuning stages. Specifically, we first introduce consistent pseudo-video augmentations to create diverse pseudo-video samples for pre-training while maintaining the instance consistency across frames. Then, we incorporate a multi-scale temporal module to enhance the model’s ability to model temporal relations through self- and cross-attention at short- and long-term temporal spans. Our approach does not set constraints on model architecture and can integrate seamlessly with various VIS methods. Experiment results on commonly adopted VIS benchmarks show that our method consistently outperforms state-of-the-art methods. Our approach achieves a notable 4.0% increase in average precision on the challenging OVIS dataset. I Introduction \textsuperscript{*} \textsuperscript{*} footnotetext: indicates corresponding author (email: [email protected]) Video Instance Segmentation (VIS) integrates classification, segmentation, and tracking of instances within a video sequence. VIS has been crucial for applications like autonomous driving [ 1 , 2 ] , video editing [ 3 , 4 ] , and video understanding [ 5 , 6 , 7 , 8 , 9 , 10 ] . VIS methods are generally categorized into two paradigms: online [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ] and offline [ 18 , 19 , 20 , 21 , 17 ] . Online methods segment video frames individually and track instances across frames, while offline approaches process the entire sequence simultaneously, integrating tracking and segmentation. Both paradigms typically follow a pre-train then fine-tune strategy. They start with an image-trained segmentation model and then apply a temporal model for instance tracking. The common pre-train then fine-tune regime can lead to sub-optimal performance due to reliance on image-trained segmentation model applied directly to videos. A key challenge in video pre-training is the limited availability of suitable datasets, as popular datasets like YoutubeVIS [ 22 , 23 ] and OVIS [ 24 ] contain only a few thousand, or sometimes just a few hundred, videos, making effective pre-training difficult. Annotating larger video datasets is costly and time-consuming. Prior studies [ 16 , 17 , 15 , 14 , 21 ] have generated pseudo-videos from static images using data augmentation, but they often fail to preserve temporal coherence. For example, CTVIS [ 16 ] generates pseudo-video by combing same image augmented by random “Copy & Paste” and “Crop Shift” independently. To address the above issue, we first introduce consistent pseudo-video generation that better simulates real instance motion by ensuring temporal consistency, particularly benefiting challenging video datasets with visually similar instances. When we generate the pseudo-video data using the proposed coherent pseudo-video generation strategy, we can train the image segmentation models to serve as a pertaining model for video instance segmentation. However, image segmentation models directly pre-trained on the pseudo-video data may compromise effectiveness due to the lack of temporal awareness. These models are inherently static and incapable of handling the dynamic nature of video sequences, where objects enter and exit the field of view. To address this, various approaches [ 18 , 19 , 20 , 21 , 17 ] have introduced temporal modules to capture instance relations over time. For instance, [ 19 ] extends the image segmentation model to video processing by introducing shared latent instance tokens. However, these models often struggle with scenarios involving intersecting temporal trajectories of multiple instances, highlighting a key limitation. In addition to the pseudo-video generation, we propose a video pre-training strategy that leverages a multi-scale temporal module (MSTM) to improve temporal processing. MSTM first captures local motion variations to enhance instance relations, then integrates these correlations across the sequence for a stronger temporal understanding. We empirically show that integrating MSTM during fine-tuning significantly boosts VIS model performance. Fig. 1: Augmentation comparison. \begin{overpic}[figures/framework.pdf] \put(10.0,11.5){\scriptsize$1^{st}$ frame feature} \put(10.0,9.0){\scriptsize$2^{nd}$ frame feature} \put(10.0,6.5){\scriptsize$3^{rd}$ frame feature} \put(10.0,4.0){\scriptsize$4^{th}$ frame feature} \put(8.0,41.0){\scriptsize Pseudo-Video ($V$)} \par\put(34.5,34.3){\scriptsize Backbone} \put(34.0,25.0){\scriptsize Transformer} \put(35.0,22.5){\scriptsize Encoder} \par\put(29.0,11.0){\scriptsize Temporal} \put(30.0,9.5){\scriptsize Swap} \par\put(29.0,14.7){\tiny$f_{s}^{1}$} \put(43.7,14.7){\tiny$f_{s}^{2}$} \put(58.6,4.5){\tiny$f_{s}$} \par\put(56.4,14.7){\tiny$Q$} \put(62.7,14.7){\tiny$K,V$} \par\put(58.0,25.5){\tiny$z_{s}^{l}$} \par\put(29.0,5.0){\tiny$f_{s}^{2}$} \put(44.1,5.0){\tiny$f_{s}^{1}$} \put(62.6,4.5){\tiny$f_{s}^{{}^{\prime}}$} \par\put(59.0,41.0){\scriptsize MSTM} \put(85.0,41.0){\scriptsize Output} \put(59.0,36.8){{\color[rgb]{1,1,1}\scriptsize FPN}} \put(57.3,29.8){{\color[rgb]{1,1,1}\scriptsize ConvGRU}} \put(57.0,22.0){\scalebox{0.8}{\scriptsize{\color[rgb]{1,1,1}Norm \& Add}}} \put(57.7,17.3){\scalebox{0.8}{\scriptsize{\color[rgb]{1,1,1}Swin Cross}}} \put(57.0,12.5){\scalebox{0.8}{\scriptsize{\color[rgb]{1,1,1}Norm \& Add}}} \put(57.7,8.0){\scalebox{0.8}{\scriptsize{\color[rgb]{1,1,1}Swin Self}}} \par\put(65.5,7.0){\scriptsize$L\times$} \put(63.8,27.5){\scriptsize$N\times$} \par\put(80.3,10.5){\scriptsize Transformer Decoder} \end{overpic} Fig. 2: Overview of VIS model integrated with multi-scale temporal module. Our contributions are as follows: 1) We introduce the first video pre-training approach for VIS, bridging the gap between image pre-training and video fine-tuning. This method is easily integrated with various VIS models. 2) We develop a consistent pseudo-video generation technique that enriches training data diversity while maintaining consistency of video content. 3) We design a multi-scale temporal module to leverage both long- and short-term temporal correlations during pre-training and fine-tuning. 4) Our framework achieves state-of-the-art performance on three standard VIS benchmarks and also improves image instance segmentation. II Related Work II-A Online Video Instance Segmentation Online approaches [ 11 , 12 , 13 , 14 , 15 , 16 , 17 ] generate instance segmentations up to the latest frame without requiring future frames. These approaches typically add a temporal module for tracking, treating segmentation, and tracking separately. Amongst them, CTVIS [ 16 ] builds on Mask2Former [ 25 ] and incorporates the insight of online methods to learn more robust instance embeddings. In contrast, our work employs consistent pseudo-video augmentation during pre-training and integrates a multi-scale temporal module to better handle temporal data. II-B Data Augmentation Various augmentation techniques have been extensively explored in images [ 26 , 27 , 28 , 29 , 30 , 31 ] to enrich data sets and improve model training. Similar strategies have been applied in VIS during fine-tuning [ 26 , 32 , 27 ] , but they would often fall short in capturing the complex dynamics of real videos. CTVIS [ 16 ] combines techniques such as copy-paste, cropping, and rotation, but using such augmentation lacks the capability to accurately simulate intricate instance interactions. To address this, we propose to enrich the pseudo-video diversity by combining various augmentation effects from a predefined pool. Our approach dynamically selects and reintegrates instances within the image, adding more realism compared to template-based methods. III Video Pre-training for VIS III-A Task Definition Formally, given an input video sequence \mathbf{V}=\{I_{t}\}_{t=1}^{T} , where I_{t} represents the t -th frame in the video and T is the total number of frames, the goal of video instance segmentation is to produce pixel-level segmentation masks for each object instance present in each frame. Let \mathbf{O}=\{o_{i}\}_{i=1}^{N} denote the set of N object instances across all frames, where o_{i} represents an individual object instance. The