Your Multimodal Speech Model Says I Have a Face for Radio
Maya K. Nachesa, Vlad Niculae, Vagrant Gautam · 2026
Read Your Multimodal Speech Model Says I Have a Face for Radio by Maya K. Nachesa, Vlad Niculae, Vagrant Gautam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30472.txt.
Introduction
Maya K. Nachesa, Vlad Niculae and Vagrant Gautam give the first bias test of multimodal speech recognition. They pair varied faces with the same audio, and track shifts in transcription accuracy. They find large service gaps across mWhisper-Flamingo and Gemini models, with drops up to about four points on the word error rate metric across self-declared gender, ethnicity, and their overlap. Developers should test, fix, and report such limits, since more signals may even bias outcomes.
Catalog record
| CREATOR | Maya K. Nachesa, Vlad Niculae, Vagrant Gautam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33940 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Your Multimodal Speech Model Says I Have a Face for Radio Maya K. Nachesa, Vlad Niculae, Vagrant Gautam Abstract As large neural models have become better at language tasks, researchers are increasingly building multi- and omnimodal models that handle more modalities of data. One example is the expansion of speech recognition models to audio-visual data for noise mitigation and multimodal subtitling. While performance and bias have been studied extensively in the single-modality regime, it is unknown how new modalities affect this, even though they produce biases in humans. We therefore propose the first bias evaluation of multimodal speech recognition, where we create videos pairing different faces with the same audio, and measure changes in speech transcription accuracy. We find large quality-of-service differences across mWhisper-Flamingo and Gemini models, with drops of up to 4.05 4.05 word error rate points, across self-declared gender, ethnicity, and their intersection. Our findings point to a priority for developers to evaluate, fix, and communicate such limitations, as providing more signals through additional modalities is not necessarily better, and may even lead to biased outcomes. 1 Introduction As large neural models improve on single modalities of language data like text or audio, it is becoming increasingly common to incorporate multiple modalities into single “multimodal” or “omnimodal” models, such as images and text ( 27 ; 20 ; 2 ) , or video and audio ( 33 ; 29 ; 30 ; 34 ) . Within audio-based research, the expansion to multimodality has focused on audio-visual speech recognition, with applications in noisy environments ( 29 ; 30 ) , automatic video captioning, subtitling ( 11 ) , and more. Auditory and visual processes are also implicated in human speech perception ( 23 ) , where it similarly helps us deal with noisy environments ( 37 ) . However, visual perceptions of the speaker can sometimes worsen our auditory processing of their speech, a phenomenon termed reverse linguistic stereotyping ( 15 ) . In other words, nonlinguistic factors such as visually-perceived speaker ethnicity affect speech perception and processing ( 31 ; 32 ; 42 ; 3 ; 18 ) . No study thus far has examined whether such multimodal raciolinguistic biases also appear in audio-visual speech recognition systems, despite a long history of study in single-modality settings in vision ( 6 ) and in speech ( 17 ) . The lack of such studies is largely due to the difficulty of collecting adequate data: Videos must be audio-matched to eliminate confounding factors, as in the matched-guise setup from sociolinguistics. We propose a synthetic setup that allows us to evaluate whether the ethnicity and gender of different faces affects audio-visual speech recognition performance at scale. We construct a dataset of 75,000 videos where English audio samples that vary in auditory characteristics (accent and noise) are systematically paired with lip-synced faces that vary in visual characteristics (ethnicity and gender). Using this dataset, we evaluate audio-visual speech recognition with two models: mWhisper-Flamingo medium ( 30 ) and Gemini- 2.5 -Flash ( 8 ) . We show that our synthetic setting is a valid testbed for bias, as our lip-synced videos help audio-visual speech recognition (AVSR) performance under noisy conditions, just like natural video. Both with and without noise, AVSR models show single-axis and intersecting ethnicity and gender biases when transcribing the exact same set of audios with different faces. Our findings thus provide evidence of reverse linguistic stereotyping in AVSR models. We hope that our work spurs developers to fix bias in contexts where biases may come from multiple modalities, towards better and fairer multimodal speech recognition. 1 1 1 Code at: github.com/ltl-uva/MultiModalSpeechBias . Bias statement. In response to calls for clearer articulations of bias in language technology ( 4 ; 39 ) , we conceptualize bias in our work as differences in quality-of-service with audio-visual speech recognition, based purely on a change in visual cues such as race and gender that should be irrelevant to transcription of the exact same audio. Our definition of bias is parallel to reverse linguistic stereotyping in humans, and we measure it by testing for statistically significant changes in word error rate (WER), which can have detrimental effects on users of speech recognition technologies ( 24 ; 22 ) . 2 Related Works Multimodal speech perception in humans. The exact same English audio gets different intelligibility ratings from listeners when paired with pictures of White versus South Asian faces ( 18 ) , or Asian faces ( 32 ) . However, this effect seems to disappear with video ( 42 ) . Notably, these works use a very small sample size (three unambiguous words, presented on an accent continuum in 42 ), whereas we test at a much larger scale with automated and computational methods. Our methods are inspired by this literature (in particular the matched-guise test setup), but we evaluate speech recognition systems rather than human listeners, and focus on transcription rather than speaker attitudes as in some of these studies. Single-modality bias in vision and speech. Prior to our work, quality-of-service differentials based on race and gender have been shown in image classification ( 6 ) , image captioning ( 13 ; 41 ) , and English speech recognition ( 38 ; 17 ) . In contrast to our work, none of these papers study multi-modal speech recognition biases. Additionally, work on bias in speech recognition tends to focus on acoustically-perceptible differences (accent, dialect, etc.), in contrast to our work, where identical audio is paired with different visual input. Multi- and omnimodal models. Analyses of multimodal models focus overwhelmingly on vision-language models that take images and text as input. For instance, 12 show that these models tend to overlook their vision input representations, 14 analyze how race, gender, and skin tone affect these models’ attribution of emotions, 5 evaluate multimodal, multicultural hate speech, and 16 evaluate models’ cultural stereotypes when visual ethnicity information is perturbed. Although this last paper’s methodology is similar to ours in terms of systematically perturbing ethnicity features, our work focuses on video and speech modalities, which none of the above consider. Closest to our work, 9 evaluate the robustness of audiovisual speech recognition to dropped frames, while we look at robustness to variance in the visual component of the input. 3 Methodology The dominant signal for speech recognition is the audio. In order to disentangle the impact of visual biases from audio-level performance differences in AVSR, we require a controlled setup with rich demographic information. No such audio-visual dataset is publicly available. Therefore, we create a usefully-close approximation through modifying and combining existing datasets. The result allows us to compare the performance of AVSR models with varying visuals (faces of varying ethnicities and genders) against the same audio signal. 3.1 Dataset To evaluate multimodal speech recognition bias, we create a dataset of videos that systematically combine faces and audio files, corresponding to experiments in the matched-guise setup with humans ( 18 ; 42 ) . Faces. For high-resolution, standardized photographs of racially diverse faces, we use the Chicago Face Database ( 21 ) and the India Face Set ( 19 ) . From these datasets, we select a total of 500 faces that are marked to have a neutral expression.