Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos
Yuchi, Shota, Hokuto, Kazuhiro, Tatsuya, Yoshimitsu · 2025
Read Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos by Yuchi, Shota, Hokuto, Kazuhiro, Tatsuya, Yoshimitsu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11967.txt.
Introduction
Yuchi Ishikawa and five co-authors teach a model to learn from sound, images, and words at once. Their paper appeared in two thousand twenty-five. Masked autoencoders learn by hiding parts of audio and video and guessing them back. Adding language guidance can sharpen what they learn. The catch is training data: matched sound, image, and text triples are scarce.
The team makes those triples on its own from raw videos. An image captioning model writes frame-level captions first. A filter based on CLAP then keeps only triples where sound and caption align well. No hand labels are needed. Their model, LG-CAV-MAE, adds a pretrained text encoder to the audio-visual setup. It beats prior methods on retrieval and classification tests. The gains show the value of clean auto-made triples.
Catalog record
| CREATOR | Yuchi, Shota, Hokuto, Kazuhiro, Tatsuya, Yoshimitsu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-11967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos Yuchi, Shota, Hokuto, Kazuhiro, Tatsuya, Yoshimitsu Abstract In this paper, we propose Language-Guided Contrastive Audio-Visual Masked Autoencoders (LG-CAV-MAE) to improve audio-visual representation learning. LG-CAV-MAE integrates a pretrained text encoder into contrastive audio-visual masked autoencoders, enabling the model to learn across audio, visual and text modalities. To train LG-CAV-MAE, we introduce an automatic method to generate audio-visual-text triplets from unlabeled videos. We first generate frame-level captions using an image captioning model and then apply CLAP-based filtering to ensure strong alignment between audio and captions. This approach yields high-quality audio-visual-text triplets without requiring manual annotations. We evaluate LG-CAV-MAE on audio-visual retrieval tasks, as well as an audio-visual classification task. Our method significantly outperforms existing approaches, achieving up to a 5.6% improvement in recall@10 for retrieval tasks and a 3.2% improvement for the classification task. Ishikawa Nakada Munakata Saito Komatsu Aoki LY CorporationJapan Keio UniversityJapan keywords audio-visual retrieval, audio-visual classification, contrastive learning, dataset generation † † email: [email protected], [email protected] 1 Introduction Understanding the relationship between audio and visual modalities is crucial for various applications, including video content analysis and autonomous systems. Existing methods [ 1 , 2 , 3 ] have sought to learn correspondences between auditory and visual signals through self-supervised audio-visual learning, achieving high performance on classification and retrieval tasks. Multimodal learning typically requires large amounts of paired data, which can be costly to collect. In terms of audio-visual pairs, videos are a relatively easy source, as they typically contain both auditory and visual modalities. To date, a variety of audio-visual datasets, including AudioSet [ 4 ] , Kinetics-Sound [ 5 ] , VGGSound [ 6 ] , and ACAV100M [ 7 ] , have been released. However, from a quality standpoint, audio data often contains significant noise (e.g., off-screen sounds, background music), making it challenging to collect audio-visual data with clear alignment. Consequently, existing state-of-the-art audio-visual learning methods, such as CAV-MAE [ 1 ] , still struggle to accurately capture fine-grained audio-visual correspondences. To address this challenge, [ 8 ] has proposed DETECLAP, an extension of CAV-MAE. DETECLAP is a learning framework that, in addition to training a contrastive audio-visual masked autoencoder, incorporates an auxiliary task of predicting object labels detected within videos. This framework enables learning audio-visual representations that focus on fine-grained objects rather than coarse object categories. However, DETECLAP relies heavily on predefined object labels, which limits its ability to learn correspondences for objects not defined by those labels or nuances beyond object-level features (e.g. intense vs. gentle guitar performance). In this paper, we propose Language-Guided CAV-MAE (LG-CAV-MAE), a simple yet effective framework for enhancing audio-visual representations. LG-CAV-MAE extends CAV-MAE by leveraging a pretrained text encoder to incorporate contrastive losses for audio-visual-text triplets. This framework enables the learning of more detailed audio-visual feature representations without relying on predefined object labels, as required by DETECLAP. Figure 1: Existing audio-visual-text data vs. our generated triplets. (left) Existing datasets often lack strong alignment between audio and vision or have missing text. (right) Our proposed triplet generation method is based on the idea that if a caption generated from a video describes the audio accurately, the audio and visual components are also likely to correspond. Therefore, this method can generate well-aligned triplets. (a) Overview of our LG-CAV-MAE). (b) Our proposed audio-visual-text triplet generation process. Figure 2 : (a) The upper part illustrates the training process of the contrastive audio-visual masked autoencoder (CAV-MAE) [ 1 ] . The lower part shows how we introduce additional audio-text and visual-text contrastive losses using a pretrained text encoder, extending CAV-MAE to a tri-modal framework (LG-CAV-MAE). (b) For LG-CAV-MAE training, we propose to automatically generate audio-visual-text triplets from unlabeled videos using an image caption models and CLAP [ 9 ] . Although training LG-CAV-MAE requires audio-visual-text triplets, most of existing datasets [ 10 , 11 , 12 , 6 , 7 ] do not provide textual descriptions nor do they offer well-aligned audio-visual pairs, primarily due to background music and off-screen sounds (the left part of Fig. 1 ). Therefore, these datasets are not suitable for our LG-CAV-MAE. To address this problem, we further propose a method for automatically generating captions from unlabeled videos and filtering out those that are not descriptive on the underlying audio (the right part of Fig. 1 ). This framework automatically constructs audio-visual-text triplets used for our LG-CAV-MAE training. Our approach is based on the hypothesis that if a caption generated from a video describes the audio accurately, the audio and visual components are also likely to correspond. Specifically, we first sample frames from unlabeled videos and employ an image captioning model to generate captions for each frame. We then compute the similarity between these generated captions and the original video’s audio using CLAP [ 9 ] , selecting instances with high similarity as audio-visual-text triplets. This procedure allows us to construct datasets with robust audio-visual alignment without explicitly comparing audio-visual correspondence as in existing methods [ 7 , 13 ] . Note that, unlike in [ 14 , 15 , 16 , 17 ] , our method does not require metadata or tags associated with audio or videos. In our experiments, using our automatic data generation method, we build audio-visual-text triplets from two existing video datasets, VGGSound [ 6 ] and Kinetics700 [ 18 ] . We then train LG-CAV-MAE with these generated triplets and evaluate the model on audio-to-visual retrieval, visual-to-audio retrieval, and audio-visual classification tasks. LG-CAV-MAE achieves performance surpassing that of existing methods without incurring any additional annotation costs. 2 Proposed Method In this paper, we propose Language-Guided CAV-MAE (LG-CAV-MAE) which is built upon CAV-MAE to enhance audio-visual representations. We also propose to automatically generate audio-visual-text triplets from unlabeled videos, which are subsequently used for training LG-CAV-MAE. In the following sections, we first provide an overview of CAV-MAE, then explain our extension of it. Finally, we detail our method for automatically generating audio-visual-text triplets from videos. 2.1 Preliminary: CAV-MAE CAV-MAE [ 1 ] is a powerful self-supervised learning framework designed to learn audio-visual representations (see the upper part of Fig. 2(a) ). Let \mathcal{D}=\{(\mathbf{X}^{a},\mathbf{X}^{v})\} be a dataset consisting of paired audio and visual samples, where \mathbf{X}^{a}\in\mathbb{R}^{T_{a}\times F} is an audio log mel spectrogram with T_{a} time steps and F frequency bins, and \mathbf{X}^{v}\in\mathbb{R}^{H\times W\times C} is a single video frame of height H , width W , and C channels. CAV-MAE aims to learn audio-visual representations by reconstructing masked regions in each modality and aligning audio and visual representations in a shared embedding space. In CAV-MAE training, we first randomly mask a subset of tokens for both the visual frame and the audio spectrogram. For the visual input, we patchify \mathbf{X}^{v} and select a subset of patches indexed by \Omega_{v} . We create a binary mask \mathbf{M}^{v} whose entries are 0 for masked patches and 1 otherwise, and obtain the masked version \widetilde{\mathbf{X}}^{v}=\mathbf{M}^{v}\odot\mathbf{X}^{v} . Similarly, we split the audio spectrogram into patches and choose a subset indexed by \Omega_{a} to mask. Let \mathbf{M}^{a} be the corresponding binary mask; we then have \widetilde{\mathbf{X}}^{a}=\mathbf{M}^{a}\odot\mathbf{X}^{a} . Next, each encoder takes the corresponding masked input. The visual encoder processes \widetilde{\mathbf{X}}^{v} and produces a latent representation \mathbf{v}=\mathrm{VisualEncoder}(\widetilde{\mathbf{X}}^{v})\in\mathbb{R}^{d} where d is the dimensionality