Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction Conference: Proceedings of the 2025 International Conference on Multimedia Retrieval; June 30-July 3, 2025; Chicago, IL, USA Proceedings of the 2025 International Conference on Multimedia Retrieval (ICMR ’25)
Zihan Zhou, Changrui Dai, Aibo Song, Xiaolin Fang · 2025
Read Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction Conference: Proceedings of the 2025 International Conference on Multimedia Retrieval; June 30-July 3, 2025; Chicago, IL, USA Proceedings of the 2025 International Conference on Multimedia Retrieval (ICMR ’25) by Zihan Zhou, Changrui Dai, Aibo Song, Xiaolin Fang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21692.txt.
Introduction
Zihan Zhou, Changrui Dai, Aibo Song, and Xiaolin Fang wrote a paper in 2025 on fine-grained video object tracking, following an object through video at the level of individual pixels. Accurate video analysis requires the same pixels to be recognized across frames. A popular self-supervised approach learns that correspondence by reconstructing frames, but earlier methods make little direct use of multiple reference frames.
Their framework, Dynamic Memory Prediction, keeps a memory bank of reference frames and dynamically selects the best ones for each new frame, based on the tracked object's pixel features. A bidirectional target prediction network uses those frames to make tracking more robust. The memory selection keeps the match accurate even when the object is occluded or moves fast. In experiments, the method outperformed the best existing self-supervised techniques on two fine-grained tasks: object segmentation and keypoint tracking.
Catalog record
| CREATOR | Zihan Zhou, Changrui Dai, Aibo Song, Xiaolin Fang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21692
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction Conference: Proceedings of the 2025 International Conference on Multimedia Retrieval; June 30-July 3, 2025; Chicago, IL, USA Proceedings of the 2025 International Conference on Multimedia Retrieval (ICMR ’25), June 30-July 3, 2025, Chicago, IL, USA DOI: 10.1145/3731715.3733327 ISBN: 979-8-4007-1877-9/2025/06 CCS: Computing methodologies Video segmentation CCS: Computing methodologies Tracking Zihan Zhou, Changrui Dai, Aibo Song, Xiaolin Fang Abstract . Successful video analysis relies on accurate recognition of pixels across frames, and frame reconstruction methods based on video correspondence learning are popular due to their efficiency. Existing frame reconstruction methods, while efficient, neglect the value of direct involvement of multiple reference frames for reconstruction and decision-making aspects, especially in complex situations such as occlusion or fast movement. In this paper, we introduce a Dynamic Memory Prediction (DMP) framework that innovatively utilizes multiple reference frames to concisely and directly enhance frame reconstruction. Its core component is a Reference Frame Memory Engine that dynamically selects frames based on object pixel features to improve tracking accuracy. In addition, a Bidirectional Target Prediction Network is built to utilize multiple reference frames to improve the robustness of the model. Through experiments, our algorithm outperforms the state-of-the-art self-supervised techniques on two fine-grained video object tracking tasks: object segmentation and keypoint tracking. 0 10^{1} 10^{2} 10^{3} 10^{4} 10^{5} 10^{6} 10^{7} 64 66 68 70 72 74 76 78 80 82 84 86 MAST ( Lai et al. 2020 ) VFS ( Xu and Wang 2021 ) CRW ( Jabri et al. 2020 ) CLTC ( Jeon et al. 2021 ) DINO ( Caron et al. 2021 ) LIIR ( Li et al. 2022 ) MASK-VOS ( Li et al. 2023a ) Ours STCN ( Cheng et al. 2021 ) EGMN ( Lu et al. 2020 ) STM ( Oh et al. 2019 ) Fasttan ( Huang et al. 2020 ) AFB-URR ( Liang et al. 2020 ) MHP-VOS ( Xu et al. 2019 ) AGAME ( Johnander et al. 2019 ) RGMP ( Oh et al. 2018 ) Number of pixel-level annotations (log scale) \mathcal{J}\&\mathcal{F}(Mean) Figure 1 . Performance comparison over DAVIS 17 ( Perazzi et al. 2016 ) val. The self-supervised method is positioned on the far left of the figure, while the supervised method is located on the right side. Ours surpasses all existing self-supervised methods ( \mathcal{J}\&\mathcal{F}(Mean) : 76.4), and is on par with many fully-supervised ones trained with massive annotations. 1. Introduction Unlike supervised video object tracking learning, self-supervised methods alleviate the need for fine pixel-level annotation of video objects and can quickly and cheaply build mature and usable models, which has significant advantages. Due to the inherent temporal and spatial consistency of natural video sequences, utilizing unlabeled natural videos for self-supervised video object tracking has become a feasible solution. Furthermore, rich natural video information enhances the generalization ability of models, potentially surpassing supervised methods in terms of information availability and applicability. It’s worth noting that the Temporal Correspondence Learning method has gained popularity through extensive practical use. However, some researchers have overlooked a fundamental issue: pixel correspondence is not limited to a single or static reference frame similar to the query frame; it spans the entire video sequence. Admittedly, some researchers have recognized that the quality of frame reconstruction can be improved by using more reference frames, which is usually achieved through methods such as temporal attention mechanisms. While this approach allows for the use of multiple reference frames, it also poses a number of challenges: loss of pixel information during secondary coding, increased training costs, and additional memory overhead. These issues undermine the practical advantages of the approach. Therefore, we design a framework that allows multiple reference frames to participate directly and efficiently in frame reconstruction. The framework encodes a sequence of video frames uniformly once (encoder only), and then enables the model to heuristically select a sufficient number of reference frames in real-time and dynamically during the reconstruction process. In order to make up for the neglect of multi-frame reference in previous works and enable the model to make full and appropriate use of reference frames, we propose a new frame reconstruction framework, including: • we propose Reference Frame Memory Engine , which manages two memory banks: a Short-Term Memory bank comprising reference frames temporally proximate to the current query frame and exhibiting high similarity to it, and a Long-Term Memory bank storing reference frames temporally distant from the query frame but containing crucial initial target pixel information for tracking. Both memories are updated dynamically. • we propose Bidirectional Target Prediction Network to handle both long-term and short-term reference frames separately. This approach through building cluster models to establish region correspondence between different frames. For short-term reference frames, we distinguish pixels of different targets through backward-to-forward matching to achieve more precise affinity calculations. For long-term reference frames, we constrain the range of candidate regions for query pixels through forward-to-backward matching, helping to suppress offset during label propagation. Overall, we propose a new framework for self-supervised video correspondence learning named DMP (Dynamic Memory Prediction). This method introduces rich tracking object pixel information through the Reference Frame Memory Engine, and makes good use of this information through the Bidirectional Target Prediction Network. This allows our framework to show excellent accuracy and robustness, surpassing other methods in the same field. And in certain cases, it shows comparable or adaptable performance to fully supervised methods, as shown in Figure 1 . 2. Related Work 2.1. Self-Supervised Temporal Correspondence Learning In the field of video processing, understanding correspondence plays a crucial role in a variety of tasks including video segmentation ( Hu et al. 2018 ) , flow estimation ( Dosovitskiy et al. 2015 ; Zhao et al. 2022 ) , and video object tracking ( Bertinetto et al. 2016 ) . Unlike traditional supervised methods, self-supervised approaches for correspondence learning have capitalized on diverse and abundant unlabeled video datasets ( Gan et al. 2018 ; Kim et al. 2019 ; Vondrick et al. 2018 ) to discover latent correspondence patterns, leading to substantial advancements. The dominant approach of self-supervised temporal correspondence learning methods aims at reconstructing the query frame from neighboring frames using inter-frame pixel similarity. ( Lai et al. 2020 ; Lai and Xie 2019 ; Vondrick et al. 2018 ) introduces a colorization proxy task, aiming to reconstruct a query frame from its adjacent frame while leveraging their temporal correspondence. Videos are advantageous due to their inherent temporal information and spatio-temporal consistency. This allows us to use the continuity of pixel information, as pixels in one frame often correspond to pixels in the next. Based on this correlation, studies like ( Lai and Xie 2019 ) have introduced reconstruction methods where each pixel in the current frame identifies its best match in the previous frame. This reconstruction method incorporates attention mechanisms. Both the current frame and the previous frame undergo feature encoding and are projected into the pixel embedding space \Phi:R^{H\times W\times 3}\rightarrow R^{h\times w\times c} , yielding a triplet for each frame I_{t} , denoted as {{Q_{t},K_{t},V_{t}}} , where Q_{t}=K_{t}=\Phi(I_{t};\theta) . During the training phase, the value V_{t} corresponds to the original reference frame, while during the inference phase, it corresponds to the