Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation

Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun · 2025

Read Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation by Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18539.txt.

Introduction

Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, and Se-Young Yun present CAV2vec, a learner for speech that reads lips as well as sound in two thousand twenty-five. Lip reading steadies speech tools in noise, but past work mostly guards the sound side. Blurred or blocked lips can hurt just as much.

A student model trains on corrupted clips and tries to match clean targets from a teacher model. Cross drills ask for clean sound from spoiled video and clean video from spoiled sound. That cross training pulls the shared code back together when one side is damaged. On robust speech benchmarks, the method lifted accuracy across many kinds of damage.

Catalog record

CREATORSungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18539

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun Abstract Audio-visual speech recognition (AVSR) incorporates auditory and visual modalities to improve recognition accuracy, particularly in noisy environments where audio-only speech systems are insufficient. While previous research has largely addressed audio disruptions, few studies have dealt with visual corruptions, e.g., lip occlusions or blurred videos, which are also detrimental. To address this real-world challenge, we propose CAV2vec, a novel self-supervised speech representation learning framework particularly designed to handle audio-visual joint corruption. CAV2vec employs a self-distillation approach with a corrupted prediction task, where the student model learns to predict clean targets, generated by the teacher model, with corrupted input frames. Specifically, we suggest a unimodal multi-task learning , which distills cross-modal knowledge and aligns the corrupted modalities, by predicting clean audio targets with corrupted videos, and clean video targets with corrupted audios. This strategy mitigates the dispersion in the representation space caused by corrupted modalities, leading to more reliable and robust audio-visual fusion. Our experiments on robust AVSR benchmarks demonstrate that the corrupted representation learning method significantly enhances recognition accuracy across generalized environments involving various types of corruption. Our code is available at https://github.com/sungnyun/cav2vec . 1 Introduction Audio-visual speech recognition (AVSR) ( Noda et al., 2015 ; Afouras et al., 2018a ; Ma et al., 2021b ; Shi et al., 2022a ; Hsu & Shi, 2022 ; Hu et al., 2023a ) represents a significant advancement in speech recognition by integrating both auditory and visual modalities to enhance performance. This multimodal integration proves particularly vital in contexts where audio-only speech recognition systems suffer from ambient or background noise, as visual speech information like lip movements significantly improves recognition capabilities ( Makino et al., 2019 ; Ren et al., 2021 ; Chen et al., 2023 ) . In this sense, previous works on AVSR have primarily focused on overcoming audio disruptions, e.g., Xu et al. (2020) training an audio enhancement sub-network, or Shi et al. (2022b) pretraining with noise-augmented audio. Nonetheless, real-world applications often encounter scenarios where video corruption is as critical as audio disturbances. For example, we may consider an outdoor interview where not only is the audio disturbed by ambient noise or traffic sound but visual cues are also intermittently occluded, either when the speaker’s hands obstruct the view of their face or a camera is out of focus (see Figure 1(a) ). AVSR models often fail to accurately recognize the utterance under these corrupted environments. (a) (b) (c) Figure 1: (a) Real-world speech recognition challenges. AVSR models suffer from maintaining robust representations under the corrupted environments and fail to recognize utterances. (b) Our corrupted representation learning strategies with multimodal and unimodal corrupted prediction tasks. (c) Speech recognition accuracy (100 - WER %), where frequency denotes the number of visual corruption events in a sequence. Our representation learning framework, CAV2vec with a unimodal strategy (U), significantly improves robustness compared to the baseline model and even outperforms the multimodal strategy (M). Despite the effectiveness of current methods in addressing audio corruptions, there is a lack of solutions for visual corruption, highlighting the necessity for a more robust approach to tackle audio-visual joint corruption in AVSR systems. Recent efforts have addressed the visual corruption using techniques such as scoring modules to assess the reliability of audio and video frames ( Hong et al., 2023 ) , or generative pipelines to reconstruct occluded face images ( Wang et al., 2024 ) . However, these methods often rely on specific architectures or external modules, which limit their applicability. Building on recent advances in audio-visual self-supervised learning that highlight the efficacy of modality-fusion representations ( Shi et al., 2022a ; Lian et al., 2023 ; Zhang et al., 2023 ) , we propose CAV2vec , a novel audio-visual speech representation learning method designed to handle jointly c orrupted a udio- v isual data. CAV2vec is trained through a corrupted prediction task , where the model learns to predict clean targets from corrupted input sequences. For this, we employ a teacher-student self-distillation framework ( Caron et al., 2021 ; Ruan et al., 2023 ) , which has proven effective in learning contextualized speech representations ( Baevski et al., 2020 ; Baevski et al., 2022 ; Shi et al., 2022b ; Zhu et al., 2024 ; Liu et al., 2024 ) without requiring architectural changes or additional modules. In this framework, corrupted sequences are fed into the student model, while the self-evolving teacher model generates the clean targets online. Within the CAV2vec representation learning framework, the corrupted prediction task can be defined in a multimodal or unimodal strategy. A multimodal strategy, inspired by the masked prediction task in AV-data2vec ( Lian et al., 2023 ) , involves corrupting both audio and video inputs to generate corrupted multimodal features, with the model learning to predict clean multimodal targets. While this approach enhances the robustness of multimodal representations, it is less capable of isolating the effects of corruption on individual modalities, as both inputs and targets contain mixed audio-visual information. Multimodal fusion is often prone to combining redundant information ( Hsu & Shi, 2022 ; Mai et al., 2023 ) , where discriminative unimodal information is ignored, thereby the multimodal prediction falls into overfitting. Previous approaches have tackled this by improving cross-modal information, such as Mai et al. (2023) applying late-fusion to filter out noisy information from unimodal features, and Hu et al. (2023b) learning a viseme-phoneme mapping ( Bear & Harvey, 2017 ) to restore corrupted phonemes through visemes, but both of them rely on the information bottleneck structure before the fusion. To address the limitations of multimodal prediction approach, we introduce CAV2vec with a unimodal multi-task learning strategy for corrupted prediction tasks, which leverages corrupted unimodal sequences to distill cross-modal knowledge. Our unimodal strategy involves predicting clean audio targets with corrupted videos and predicting clean video targets with corrupted audios. In AVSR, this cross-modal alignment ( Ren et al., 2021 ; Hu et al., 2023b ; Hu et al., 2023c ) is essential for effectively integrating information from both modalities. Our unimodal prediction strategy, as depicted in Figure 1(b) , improves cross-modal alignment by reducing the dispersion in representation caused by the corrupted inputs. Figure 1(c) describes the AVSR performance under audio-visual jointly corrupted environments. While fine-tuning with corrupted data (CFT) improves robustness to some extent, it remains challenging at higher degrees of corruption. By incorporating the corrupted representation learning before CFT, CAV2vec demonstrates superior performance. CAV2vec with unimodal multi-task learning further enhances the corrupted prediction framework, effectively aligning the corrupted modalities and achieving more robust results. We summarize our contributions as follows. \circ We propose a novel audio-visual speech representation learning, CAV2vec, specifically designed for robustness under audio-visual joint corruption within a self-distillation framework. \circ CAV2vec conducts a unimodal multi-task learning for corrupted prediction tasks, predicting clean targets from corrupted input sequences. This strategy effectively enhances cross-modal alignment between corrupted audio and video for reliable multimodal fusion. \circ We establish an AVSR benchmark for generalization so that our setup includes novel types of corruption unseen during training, e.g., mouth occlusion by hands or face pixelation along with public noise, allowing for diverse assessment of model robustness to

Read the complete text