Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking

Yaozong Zheng, Bineng Zhong, Qihua Liang, Ning Li, Shuxiang Song · 2025

Read Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking by Yaozong Zheng, Bineng Zhong, Qihua Liang, Ning Li, Shuxiang Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21606.txt.

Introduction

Yaozong Zheng and co-authors present a tracking model that learns from unlabeled video, in a paper from 2025. Visual tracking usually needs boxes drawn by hand around targets in many frames. That labeling is costly and limits data size and variety.

Their model, called SSTrack++, trains with a weak-to-strong scheme that narrows the gap between the few labeled frames and the many unlabeled ones. Space and time consistency are taught apart: global position cues come from single frames, and motion links come from pairs. A self-prompting module tracks how the target's look changes, at both feature and decision levels. A contrast loss teaches which views show the same object. The design aims for strong tracking with almost no hand labels.

Catalog record

CREATORYaozong Zheng, Bineng Zhong, Qihua Liang, Ning Li, Shuxiang Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-21606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking Yaozong Zheng, Bineng Zhong, Qihua Liang, Ning Li, Shuxiang Song Abstract The success of visual tracking has been largely driven by datasets with manual box annotations. However, these box annotations require tremendous human effort, limiting the scale and diversity of existing tracking datasets. In this work, we present a novel Self-Supervised Tracking framework named SSTrack , designed to eliminate the need of box annotations. Specifically, a decoupled spatio-temporal consistency training framework is proposed to learn rich target information across timestamps through global spatial localization and local temporal association. This allows for the simulation of appearance and motion variations of instances in real-world scenarios. Furthermore, an instance contrastive loss is designed to learn instance-level correspondences from a multi-view perspective, offering robust instance supervision without additional labels. This new design paradigm enables SSTrack to effectively learn generic tracking representations in a self-supervised manner, while reducing reliance on extensive box annotations. Extensive experiments on nine benchmark datasets demonstrate that SSTrack surpasses SOTA self-supervised tracking methods, achieving an improvement of more than 25.3%, 20.4%, and 14.8% in AUC (AO) score on the GOT10K, LaSOT, TrackingNet datasets, respectively. Code — https://github.com/GXNU-ZhongLab/SSTrack Introduction Figure 1: The annotation requirements of different tracking tasks and our proposed framework. (a) The fully-supervised tracking methods ( Li et al. 2019 ; Chen et al. 2021 ) with frame-wise annotations. (b) The self-supervised tracking methods ( Sio et al. 2020 ; Yuan et al. 2020 ) with one-frame annotation. (c) Our self-supervised tracking method based on decoupled spatio-temporal consistency training framework and instance contrastive loss. Given an arbitrarily initial target, visual object tracking (VOT) requires recognizing and tracking an object in subsequent video frames. To accomplish this computer vision task, current high-performance visual tracking algorithms are typically trained using the full bounding box annotations of published tracking datasets ( Fan et al. 2019 ; Müller et al. 2018 ; Lin et al. 2014 ; Huang, Zhao, and Huang 2021 ) , as shown in Fig. 1 (a). However, the bounding boxes in existing VOT benchmarks rely on tremendous human efforts, making it difficult to expand their scale and diversity, such as the number of arbitrary tracked objects and open tracking scenarios. This poses a challenge for transformer-based tracking algorithms ( Ye et al. 2022 ; Xing et al. 2023 ; Zheng et al. 2024 ; Xu et al. 2025 ) , as they tend to be particularly data-hungry. From this perspective, equipping a model with the ability to automatically learn to track instances from unlabeled videos becomes crucial in the field of visual tracking. Therefore, as shown in Fig. 1 (b), we reconsider the need for box annotations by exploring a new self-supervised tracking algorithm under an initial bounding box setting . To minimize the reliance on box annotations, some self-supervised tracking methods ( Wang, Jabri, and Efros 2019 ; Sio et al. 2020 ; Yuan et al. 2020 ; Li et al. 2023 ) have been proposed to learn object correspondences from unlabeled videos. They learn instance tracking representations through contrastive learning or cycle-consistency matching strategies. For instance, S^{2} SiamFC ( Sio et al. 2020 ) and TADS ( Li et al. 2023 ) , based on supervised tracking methods, generate training pairs through random region selection and data augmentation to train self-supervised trackers. Meanwhile, self-SDCT ( Yuan et al. 2020 ) and CycleSiam ( Yuan, Wang, and Chen 2020 ) rely on the principle of cyclic consistency to construct a self-supervised tracking framework with forward-backward alignment. Despite previous studies performing well in most tracking scenarios, they still face a significant performance bottleneck due to the difficulty in effectively leveraging the rich spatio-temporal context and instance correspondence in continuous video frames . In this work, we propose a novel self-supervised visual tracking framework, called SSTrack , which aims to eliminate the need for expensive manual annotations while efficiently injecting spatio-temporal contextual information. As shown in Fig. 1 (c), we reconsider the design of the self-supervised tracking framework from a new perspective. Unlike fully supervised methods ( Cai, Liu, and Wang 2024 ; Zheng et al. 2024 ; Bai et al. 2024 ) that capture context through multi-frame inputs, directly learning temporal context in self-supervised tracking is a significant challenge due to the lack of annotated video data. To address this challenge, a decoupled spatio-temporal consistency training framework is introduced to automatically learn rich target information across timestamps. Specifically, we first perform forward tracking, globally searching for the spatial position of object. Then, we conduct backward tracking, locally perceiving the appearance and motion (temporal) changes of the instance. Through this decoupled learning approach, we achieve global spatial localization and local temporal association within a unified framework, thereby effectively utilizing both labeled and unlabeled video data. Furthermore, we introduce an instance contrastive loss function to learn instance-level correspondence across views, providing robust instance supervision without any labels. This new design paradigm enables SSTrack to effectively learn generic tracking representations in a self-supervised manner, while reducing reliance on extensive box annotations. Extensive experiments show that our approach achieves excellent tracking performance with limited annotations and significantly narrows the performance gap between self- and fully supervised tracking methods. The main contributions of this work are as follows. • We propose a novel self-supervised tracking pipeline named SSTrack, based on a decoupled spatio-temporal consistency training framework. It end-to-end learns cross-frame target representations via global spatial localization and local temporal association. • We introduce an instance contrastive loss function to learn instance-level correspondence from a multi-view perspective, offering robust instance supervision without any labels. • Our tracker achieves a new SOTA tracking results on nine visual tracking benchmarks, including GOT10K, LaSOT, TrackingNet, LaSOT ext , VOT2020, TNL2K, VOT2018, UAV123 and OTB100. Related Work Fully-Supervised Tracking Methods The prevailing visual tracking algorithms ( Bertinetto et al. 2016 ; Chen et al. 2020 ; Yan et al. 2021 ; Zheng et al. 2022 ; Zheng et al. 2023 ) predominantly adhere to the supervised tracking paradigm and achieve high performance by training on large-scale labeled datasets such as LaSOT ( Fan et al. 2019 ) , TrackingNet ( Müller et al. 2018 ) , COCO ( Lin et al. 2014 ) , and GOT10K ( Huang, Zhao, and Huang 2021 ) . These supervised tracking algorithms can be broadly categorized into two types: Siamese tracking framework ( Bertinetto et al. 2016 ; Li et al. 2018 ; Chen et al. 2020 ; Cheng et al. 2021 ; Zhang and Peng 2019 ) and Transformer tracking framework ( Ye et al. 2022 ; Cui et al. 2022 ; Chen et al. 2022 ; Xing et al. 2023 ; Zheng et al. 2024 ) . The former typically follows a three-stage approach involving feature extraction, fusion, and bounding box prediction for visual tracking, while the latter generally employs a transformer network to simultaneously perform feature extraction and fusion. Benefiting from training datasets with thousands of manual bounding box annotations, these methods have achieved significant performance gains. However, constructing large-scale video datasets is exceedingly time-consuming and costly, making it challenging to keep pace with rapid advancements in supervised tracking algorithms and often leading to gaps between data distributions in real-world scenarios. Essentially, the availability

Read the complete text