Uncovering the Visual Contribution in Audio-Visual Speech Recognition

Zhaofeng Lin, Naomi Harte · 2024

Read Uncovering the Visual Contribution in Audio-Visual Speech Recognition by Zhaofeng Lin, Naomi Harte (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17129.txt.

Introduction

Zhaofeng Lin and Naomi Harte ask how much the sight of a speaker's face helps speech tools in 2024. Tools that use both sound and video do better in noise than sound only tools. Yet it was not clear how much of that gain comes from the video. The pair test three known systems.

They measure the gain as an equal boost in sound clarity. They also look at when in a word the video helps most. Low error rates did not always mean high gains from video. Their results suggest that current tools leave much visual detail unused. They urge future work to report the visual gain next to error rates.

Catalog record

CREATORZhaofeng Lin, Naomi Harte
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17129

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncovering the Visual Contribution in Audio-Visual Speech Recognition Thanks: This work was conducted with the financial support of the Research Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real) under Grant No. 18/CRT/6224. Zhaofeng Lin, Naomi Harte Abstract Audio-Visual Speech Recognition (AVSR) combines auditory and visual speech cues to enhance the accuracy and robustness of speech recognition systems. Recent advancements in AVSR have improved performance in noisy environments compared to audio-only counterparts. However, the true extent of the visual contribution, and whether AVSR systems fully exploit the available cues in the visual domain, remains unclear. This paper assesses AVSR systems from a different perspective, by considering human speech perception. We use three systems: Auto-AVSR, AVEC and AV-RelScore. We first quantify the visual contribution using effective SNR gains at 0 dB and then investigate the use of visual information in terms of its temporal distribution and word-level informativeness. We show that low WER does not guarantee high SNR gains. Our results suggest that current methods do not fully exploit visual information, and we recommend future research to report effective SNR gains alongside WERs. I Introduction Humans perceive speech in a multimodal manner; we watch and listen when we communicate with people [ 1 ] . Viewing a speaker’s facial and lip movements enhances speech intelligibility and the contribution of visual information to speech perception becomes even more significant in challenging environments, such as those with background noise [ 2 , 3 ] . Audio-Visual Speech Recognition (AVSR) systems exploit visual information from the speaker’s face, e.g. lip movements, to enhance speech recognition. This area has attracted significant research attention in recent years. The rapid development of artificial intelligence and end-to-end modelling for Automatic Speech Recognition [ 4 ] has led to significant progress in AVSR, resulting in improved performance compared to audio-only systems across various acoustic environments [ 5 , 6 , 7 ] . Despite these advancements, the development of AVSR seems to have reached a “bottleneck” stage. State-of-the-art (SOTA) models continue to demonstrate improved performance, i.e. lower word error rates (WERs) and greater robustness, compared to their audio-only counterparts in noise. However, much of the current research remains focused on metric-driven improvements, i.e. incremental reductions to WER, with less attention given to understanding how visual information enhances these systems or whether the visual component is being fully exploited in existing approaches. This paper takes a step back to assess SOTA systems from a different perspective by considering human speech perception. Through this, we hope to gain insights as to whether the visual component is being fully exploited in existing AVSR systems, helping to uncover better approaches. Inspired by human speech perception studies, we explore this with three questions: 1) Can metrics beyond WER better reflect visual contributions? 2) How does the presence and timing of visual information influence AVSR systems? 3) Do AVSR systems perform better for words with higher levels of visual informativeness, as determined by human speech perception? Using three SOTA AVSR systems, we revisit effective SNR gains [ 8 ] to evaluate visual gains. We also present two experiments. The first occludes the mouth region at different stages of words to assess the impact of lip movements. A second experiment compares errors made by audio-only, video-only, and AVSR systems with those made in human audiovisual speech perception. The key contributions of this paper are: • We quantify the visual contribution of AVSR systems using effective SNR gains. • We reveal a discrepancy in how AVSR systems use temporal visual information compared to human reliance on early visual cues. • Our empirical investigation of the relation between AVSR errors and visual informativeness for words. II Related Research II-A Audio-visual Speech Perception In this section, we explore established contributions of visual information to speech perception, examining how temporal distribution and visual informativeness have been studied in both human perception and AVSR systems. These insights motivate our alternate approach to assessing AVSR systems. II-A 1 Quantifying Visual Contribution Many studies have attempted to quantify the contribution of visual information to speech perception [ 3 , 9 , 8 ] . MacLeod and Summerfield [ 3 ] measured auditory and audio-visual speech-reception thresholds (SRTs) in human listeners. The SRT represents the signal-to-noise ratio (SNR) in dB at which participants correctly recognise all three keywords in a speech task. The visual benefit was quantified as the difference in dB between the SRTs obtained in auditory-only and audio-visual conditions. In a 2001 study, Potamianos et al. [ 8 ] compared AVSR systems with human perception of audio-visual speech. They quantified the visual benefit in AVSR as an effective SNR gain, measured by the difference in SNR at which the AVSR WER equals the reference WER for audio-only recognition at 10 dB. They observed approximately a 6 dB effective SNR gain compared to the audio-only performance at 10 dB for both machines and humans. These findings provide a useful benchmark for our work to evaluate how effectively current AVSR systems are leveraging visual information compared to human performance. II-A 2 Temporal Distribution of Information The distribution of information in audio-visual speech varies temporally across phonemes and words. Jesse and Massaro [ 10 ] found that the timing of information availability differs, with visual speech cues often fully accessible earlier in the phoneme and words, while auditory information accumulates over time. This early visual information, although supplementary to auditory cues, provides significant early recognition benefits, particularly for visually defined linguistic features like the place of articulation. Consequently, visual speech contributes more to recognition at the beginning of a word than toward its end. Karas et al. [ 11 ] further demonstrated that visual speech can suppress auditory responses, especially for words with a visual head start. For example, in an audiovisual recording of the word ‘drive’ the visual onset of the open mouth for the initial phoneme ‘d’ precedes auditory vocalisation by 400 ms, indicating an auditory-visual asynchrony. These findings highlight the importance of timing in visual speech perception and the difference in visual information within a word. In this paper, we investigate the impact of the absence of lip movement at different stages of speech on AVSR, taking into account the early visual benefits. II-A 3 MaFI Score Overview Krason et al. [ 12 ] introduced the concept of Mouth and Facial Informativeness (MaFI) for words, quantifying visual informativeness based on mouth and facial movements. MaFI scores are derived from a speechreading task where participants watch silent video clips and guess the words being spoken. The process of computing MaFI scores involves three main steps. First, both the target words and participants’ guesses are converted into International Phonetic Alphabet (IPA) segments. Next, these IPA segments are transformed into phonological feature vectors. Finally, the similarity between the phonological features of the IPA segments is calculated as MaFI scores. The resulting scores range from approximately -2.5 to 0, with higher scores (closer to 0) indicating greater visual informativeness. MaFI scores were shown to capture the dynamic nature of mouth and facial movements well [ 12 ] , with words containing phonemes featuring roundness, frontness, or visemes like lower lip tuck, lip rounding, and lip closure being more visually informative. In this paper, we use MaFI

Read the complete text