Efficient Self-Supervised Video Hashing with Selective State Spaces
Jinpeng Wang, Niu Lian, Jun Li, Yuting Wang, Yan Feng, Bin Chen, Yongbing Zhang, Shu-Tao Xia · 2024
Read Efficient Self-Supervised Video Hashing with Selective State Spaces by Jinpeng Wang, Niu Lian, Jun Li, Yuting Wang, Yan Feng, Bin Chen, Yongbing Zhang, Shu-Tao Xia (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14518.txt.
Introduction
Jinpeng Wang and co-authors present a 2024 paper on fast video search with hashing. Hashing turns each video into a short code, so similar videos can be found quickly in huge libraries. Transformer models lead this field but cost heavy compute and memory.
The authors build on Mamba, a state-space model that scans data in a selective, data-driven way with linear cost. Their model, S5VH, uses two-way Mamba layers in both its encoder and decoder. Training pulls video features toward shared hash centers that mark meaning, with a loss that aligns the two. This learning scheme converged faster and better than prior plans. Wide tests showed gains in accuracy, transfer to new data, and search speed.
Catalog record
| CREATOR | Jinpeng Wang, Niu Lian, Jun Li, Yuting Wang, Yan Feng, Bin Chen, Yongbing Zhang, Shu-Tao Xia |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14518
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Self-Supervised Video Hashing with Selective State Spaces Jinpeng Wang, Niu Lian, Jun Li, Yuting Wang, Yan Feng, Bin Chen, Yongbing Zhang, Shu-Tao Xia Abstract Self-supervised video hashing (SSVH) is a practical task in video indexing and retrieval. Although Transformers are predominant in SSVH for their impressive temporal modeling capabilities, they often suffer from computational and memory inefficiencies. Drawing inspiration from Mamba, an advanced state-space model, we explore its potential in SSVH to achieve a better balance between efficacy and efficiency. We introduce S5VH, a Mamba-based video hashing model with an improved self-supervised learning paradigm. Specifically, we design bidirectional Mamba layers for both the encoder and decoder, which are effective and efficient in capturing temporal relationships thanks to the data-dependent selective scanning mechanism with linear complexity. In our learning strategy, we transform global semantics in the feature space into semantically consistent and discriminative hash centers, followed by a center alignment loss as a global learning signal. Our self-local-global (SLG) paradigm significantly improves learning efficiency, leading to faster and better convergence. Extensive experiments demonstrate S5VH’s improvements over state-of-the-art methods, superior transferability, and scalable advantages in inference efficiency. 1 Tsinghua Shenzhen International Graduate School, Tsinghua University 2 Harbin Institute of Technology, Shenzhen 3 Research Center of Artificial Intelligence, Peng Cheng Laboratory 4 Meituan, Beijing [email protected], {220110904,220110924}@stu.hit.edu.cn, [email protected], [email protected], [email protected], [email protected], [email protected] Code — https://github.com/gimpong/AAAI25-S5VH Introduction Figure 1: Highlights: ( a ) Our S5VH based on Mamba exhibits lower inference overheads on memory and computation. The efficiency advantages are scalable and more notable under larger frame numbers. ( b ) The introduced global learning signal in the hash space effectively enhances training efficiency, showing faster and better convergence. Content-based retrieval is a basic component in video search and recommendation. Hashing has been widely explored in this context to facilitate fast retrieval and reduce memory footprint ( Gao et al. 2023 ; Sun et al. 2023b ; Sun et al. 2024 ; Wang et al. 2024b ) . Video hashing has evolved from traditional methods with handcrafted features to advanced deep approaches with substantially improved retrieval performance, where self-supervised video hashing (SSVH) has gained increasing attention, given the ubiquity of large-scale unlabeled video data and evoked by the rapid progress of self-supervised learning in recent years ( He et al. 2020 ) . In SSVH, temporal modeling is essential for video understanding and hash code learning. Early approaches ( Song et al. 2018a ; Li et al. 2019a ) utilized nonlinear RNNs to process frame features, which suffered from gradient vanishing (or exploding) and struggled over long-range dependencies. In contrast, Transformer-based models ( Li et al. 2021 ; Wang et al. 2023a ) were shown to capture temporal semantics better, thanks to self-attentive interactions across all frames. Despite state-of-the-art retrieval performance, the quadratically growing complexity of Transformers in computation and memory to the frame number renders scalability limitations. Pursuing the optimal trade-off between efficacy and efficiency remains an under-explored problem. Recent advances have sparked interest in state space models (SSMs) ( Gu et al. 2021 ; Gupta, Gu, and Berant 2022 ) , where we find Mamba ( Gu and Dao 2023 ) , an improved variant of structured SSMs (S4) ( Gu, Goel, and Ré 2021 ) , can bring insightful solutions to the above problem. In detail, the data-dependent selective mechanism in Mamba ensures focusing on essential information while filtering out irrelevant noises regarding the input, which helps to capture temporal dynamics and understand long-range relations in videos. Moreover, Mamba’s linear complexity regarding sequence length promises superior scalability for video processing, and its GPU-friendly implementation also aligns with SSVH’s pursuit of efficient inference. These advantages motivate us to explore Mamba’s potential for SSVH. In this paper, we introduce S5VH 1 1 1 Abbr. of S elf- S upervised S elective S tate- S pace V ideo H ashing ( SSSSSVH ). , the first selective state-space approach for SSVH. For network design , we explore a Mamba-based hashing network , where each encoder or decoder layer comprises a bidirectional Mamba module for effective and efficient temporal modeling. For self-supervised hash learning , we develop a self-local-global (SLG) paradigm that excavates hierarchical learning signals to enhance training efficiency. Our motivation stems from the Mamba’s serial nature that restricts training throughout ( Liu et al. 2024 ) , and we resort to a practical and hashing-oriented solution by maximizing sample efficiency. State-of-the-art approaches ( Wang et al. 2023a ; Wei et al. 2023 ; Li, Tian, and Ng 2024 ) typically follow the intra - and inter -sample learning paradigm, where intra-sample ( i.e. , self ) signals refer to recovery tasks with various data augmentations for video understanding, while the inter-sample signals refer to contrastive tasks between videos to obtain discriminative hash codes. We note that inter-sample signals are subject to local signals reflected by individual samples, whose efficiency is limited by the (negative) sampling. We further integrate a global signal in the Hamming space to complement the learning paradigm. We start with clustering video feature space to summarize global semantic structure by cluster-level similarity. Then, we introduce a novel hash center generation algorithm to transform the global semantics into well-separated and semantically consistent hash centers. Based on them, we propose a center alignment loss to align the hash codes of each training sample to its associated hash center, supplementing global semantic guidance and enhancing sample efficiency. We conduct extensive experiments on 4 datasets: ActivityNet, FCVID, UCF101, and HMDB51, demonstrating that S5VH outperforms state-of-the-art baselines under various setups and transfers better across datasets. Regarding inference efficiency, S5VH exhibits notable advantages, including lower memory overhead, which allows for larger batchsizes, and faster computation. As shown in Figure 1 (a), these advantages become more pronounced as the frame number increases. Additionally, we provide comprehensive ablations and analyses, focusing on network architecture and training strategy. The results verify Mamba’s superiority in SSVH and confirm the necessity of the global signal in the SLG learning paradigm. In particular, Figure 1 (b) shows that the proposed center alignment loss, which serves as the global signal, can guide the training process toward faster and better convergence. To sum up, our paper makes the following contributions. • We explore the first Mamba-based SSVH model, indicating a superior solution for both efficacy and efficiency. • We design a hash center generation algorithm that computes semantically consistent and discriminative hash centers from the feature-space global semantics. • We propose a center alignment loss as a global learning signal, contributing to a solid self-local-global (SLG) paradigm and improving training efficiency. Related Works Self-Supervised Video Hashing Video hashing focuses on learning binary codes to enable fast, memory-efficient video retrieval. Self-supervised video hashing (SSVH) is particularly valuable for applications where labels are scarce. Early approaches often overlooked the temporal dynamics of videos, relying on traditional techniques like ITQ ( Gong et al. 2012 ) , SH ( Weiss, Torralba, and Fergus 2008 ) , and MFH ( Song et al. 2011 ) . VHDT ( Ye et al. 2013 ) addressed this gap and showed notable improvement. Recent methods focused on deep models, with progress in network design and learning strategy. While RNNs prevailed ( Zhang et al. 2016 ; Li et al. 2017 ;