SAVE: S peech- A ware V ideo R e presentation Learning for Video-Text Retrieval
Ruixiang Zhao Zhihao Xu Bangxiang Lan Zijie Xin Jingyu Liu Xirong Li Renmin University of China https://github.com/ruc-aimc-lab/SAVE · 2026
Read SAVE: S peech- A ware V ideo R e presentation Learning for Video-Text Retrieval by Ruixiang Zhao Zhihao Xu Bangxiang Lan Zijie Xin Jingyu Liu Xirong Li Renmin University of China https://github.com/ruc-aimc-lab/SAVE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08224.txt.
Introduction
Zhao and colleagues present SAVE, a speech-aware video representation learning method for video-text retrieval. CLIP has become the default choice for retrieval, but it provides only image and text encoders, so the dominant paradigm ignores video soundtracks entirely. Attempts to reintroduce audio by adding an audio encoder and fusing its output with visual features face two challenges: weak representation of speech content and suboptimal vision-audio fusion. SAVE answers with a dedicated speech branch for better speech embeddings and soft-ALBEF for early vision-audio alignment.
Catalog record
| CREATOR | Ruixiang Zhao Zhihao Xu Bangxiang Lan Zijie Xin Jingyu Liu Xirong Li Renmin University of China https://github.com/ruc-aimc-lab/SAVE |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37701 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08224
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAVE: S peech- A ware V ideo R e presentation Learning for Video-Text Retrieval Ruixiang Zhao Zhihao Xu Bangxiang Lan Zijie Xin Jingyu Liu Xirong Li Renmin University of China https://github.com/ruc-aimc-lab/SAVE Abstract For video-text retrieval, the use of CLIP has been a de facto choice. Since CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ignores the sound track of videos. While several attempts have been made to reintroduce audio – typically by incorporating an audio encoder and fusing its output with visual features – these methods face two challenges: ineffective representation of speech content and suboptimal vision-audio fusion . To address these issues jointly, we propose SAVE , a S peech A ware V ideo r E presentation learning method. SAVE improves upon AVIGATE, a SOTA audiovisual method, with a dedicated speech branch for more effective speech embedding. Furthermore, we introduce soft-ALBEF for early vision-audio alignment that facilitates fusion. Extensive experiments on five benchmarks show that SAVE compares favorably against the SOTA, outperforming AVIGATE by +4.1% on MSRVTT-9k , +1.9% on MSRVTT-7k , +2.5% on VATEX, +9.8% on Charades, and +2.1% on LSMDC, in light of the SumR metric. 1 Introduction (a) Limitation of current audio encoders (b) Proposed SAVE method for speech-aware video representation learning (c) Comparing with SOTA audio-enhanced models Figure 1 : An overview of this paper . (a) Problem : Current audio encoders ( ResNet-18 13 and AST 9 ), trained on datasets of environmental sounds, are not well suited for speech embedding. (b) Solution : We improve the state-of-the-art in audio-enhanced video-text retrieval by introducing a dedicated speech branch for speech-aware video embedding. (c) Results : Our method consistently outperforms SOTA audio-enhanced models (TEFAL 17 and AVIGATE 18 ) across five benchmarks. Video-text retrieval (VTR), a fundamental task in vision-language understanding, aims to retrieve the most semantically relevant video w.r.t. a given text query or vice versa. Owing to the remarkable success of Contrastive Language-Image Pre-training (CLIP) 35 in bridging visual and textual modalities, the use of CLIP has been a de facto choice for VTR 32 ; 50 ; 10 ; 45 ; 23 . Since CLIP provides only image and text encoders, these CLIP-based methods naturally neglect the sound track of videos. In order to reintroduce the sound track, several attempts have been made 27 ; 17 ; 18 , see Table 1 . They typically place an audio encoder alongside the image encoder for audio embedding. Subsequently, audio-enhanced video embedding is obtained by fusing the audio embedding with its visual counterpart from the vision encoder. Concerning the choice of vision-audio fusion, text-conditioned attention is considered in TEFAL 17 , whilst symmetric and asymmetric cross-attention based fusions are used in EclipSE 27 and AVIGATE 18 , respectively. These audiovisual methods have demonstrated superior performance compared to their vision-only counterparts. Table 1 : SOTA audiovisual methods for video-text retrieval . Method Audio Speech Video embedding network Early vision-audio alignment MV-9k SumR EclipSE 27 ResNet-18 ✗ Bi-branch ✗ 197.8 TEFAL 17 AST ✗ Bi-branch ✗ 209.2 AVIGATE 18 AST ✗ Bi-branch ✗ 207.7 SAVE (ours) AST ASR + CLIP Tri-branch ✓ 216.2 With a closer look at the audio encoders currently used, i.e . ResNet-18 13 in EclipSE and AST 9 in TEFAL and AVIGATE, we notice that they are not well suited for speech embedding. The reason is surprisingly simple: the audio encoders were not trained for speech content understanding. To demonstrate such a deficiency, we design a toy experiment to inspect whether existing audio encoders can produce well-separated embeddings for speech content. For three animal classes, i.e . bird , cat , and dog , we construct three modality-specific datasets: (i) Sound : Real animal vocalizations collected from AudioSet 7 , (ii) Speech : Narration clips 1 1 1 We first prompt an LLM to generate text descriptions about the characteristics per animal, and then use a Text-to-Speech model to convert these texts into narration clips. describing characteristics per animal, and (iii) Text : Transcripts obtained by performing Automatic Speech Recognition (ASR) with Whisper 36 on the Speech dataset. Each dataset contains 100 samples per class. Sound and speech embeddings are acquired with AST, while text embeddings are produced by the CLIP text encoder. As shown in Fig. 1(a) , while the sound and text embeddings are well separated class-wisely, the speech samples are cluttered in the audio feature space 2 2 2 Similar cluttered results are also observed in the audio feature space of Whisper, see the supplement, though it has good ASR performance. . Figure 2 : Hard vs . soft labels for early vision-audio alignment . For the videos in the first two rows, their associated sound tracks are not semantically relevant w.r.t. the video content. Enforcing vision-audio alignment for these videos is adverse. By contrast, the soft labels, estimated by ImageBind 8 , provide finer supervision for better vision-audio alignment. Another overlooked issue in the current audiovisual methods is the difficulty in vision-audio fusion. Consider AVIGATE for instance. The method fully relies on a cross-attention based module to directly fuse the frame features (produced by the CLIP image encoder) and the audio features (produced by AST), which are misaligned by definition. Meanwhile, lessons learned in the context of vision-language pre-training show that align before fuse (ALBEF) 25 helps. That said, directly applying ALBEF for the vision-audio alignment is problematic. Unlike image-caption pairs used for vision-language pre-training, vision-audio pairs often lack semantic correspondence, see Fig. 2 . Consequently, enforcing early vision-audio alignment on the noisy pairs with hard labels may cause the model to learn spurious correlations rather than genuinely meaningful alignment. In order to jointly address the two issues, we propose in this paper a S peech A ware V ideo r E presentation learning method ( SAVE ). More specifically, for effective speech embedding, we extend the AVIGATE framework with a dedicated speech branch. Meanwhile, we develop soft-ALBEF for early vision-audio alignment, harnessing ImageBind 8 to provide soft, noise-tolerant supervision. To sum up, our main contributions are three-fold: \bullet The first speech-aware video embedding for CLIP-based VTR . We introduce a dedicated speech branch that converts narration audio into text via ASR and encodes it with a text encoder, enabling CLIP-based retrieval models to explicitly capture semantic speech information that traditional audio encoders fail to represent. \bullet Soft-ALBEF for robust early vision-audio alignment . We propose a noise-tolerant alignment strategy that leverages ImageBind to generate soft supervision signals, effectively mitigating the semantic mismatch and spurious correlations caused by directly enforcing hard alignment on noisy vision-audio pairs. \bullet New state-of-the-art . SAVE achieves superior performance across five VTR benchmarks, outperforming AVIGATE by +4.1% on MSRVTT-9k , +1.9% on MSRVTT-7k , +2.5% on VATEX, +9.8% on Charades, and +2.1% on LSMDC, in light of the SumR metric. 2 Related Work This work aims to learn speech-aware video representation for VTR. Accordingly, we review three relevant lines of research: CLIP-based VTR, audio-enhanced VTR, and the use of speech for VTR. CLIP-based VTR . CLIP4Clip 32 is the first to adopt CLIP for end-to-end VTR. More recent models improve the retrieval performance by introducing fine-grained interactions between the vision and text modalities, including