RedVox: Safety and Fairness Gaps in Speech Models Across Languages

Beatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, Luisa Bentivogli · 2026

Read RedVox: Safety and Fairness Gaps in Speech Models Across Languages by Beatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, Luisa Bentivogli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26968.txt.

Introduction

Beatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, and Luisa Bentivogli expose safety gaps in speech models outside English. Voice-enabled models now serve real users in many languages. Their makers rarely test safety beyond English, the authors find: only a small share of releases document any multilingual safety analysis. Their benchmark, RedVox, fills the gap with real voices.

RedVox covers unsafe and stereotyped requests in five languages. Eight leading models were evaluated. Weaknesses appeared even without adversarial tricks. They worsened in non-English languages and grew again when requests were spoken rather than typed. The team also surveyed the volunteers who recorded voices. Their answers document privacy and personal burdens unique to collecting speech data for safety research.

Catalog record

CREATORBeatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, Luisa Bentivogli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RedVox: Safety and Fairness Gaps in Speech Models Across Languages Beatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, Luisa Bentivogli Abstract Speech-capable models are increasingly deployed in real-world applications across languages. Yet their safety and fairness beyond English settings and under naturalistic conditions remain understudied. We survey safety reporting practices across state-of-the-art speech model releases, finding that only 8% document any multilingual analysis. To address this gap, we introduce RedVox , a multilingual safety and fairness benchmark for audio and speech built on real voices, covering unsafe and unfair stereotypical requests across five languages (English, French, Italian, Spanish, and German). Evaluating eight state-of-the-art models, we find that vulnerabilities persist even under non-adversarial conditions, worsen in non-English languages, and are amplified when the request comes from a spoken input. Finally, by surveying the participants who contributed to RedVox , we document the unique personal and privacy challenges of collecting speech data with human participants, pointing to broader sociotechnical challenges in naturalistic speech safety research. Warning : This paper contains examples of harmful language. 1 Introduction The integration of speech into large language models (LLMs) marks a frontier advancement for human-AI interaction. Unlike traditional pipelined architectures, recent speech-capable models support direct spoken interaction by processing input audio natively ( Gaido et al. 2024 ; Nguyen et al. 2025 ; Qwen Team 2026 , e.g.,) —thereby preserving salient paralinguistic features—and offer practical advantages such as reduced latency and inherent error resilience ( Ji et al. 2024 ) . These capabilities unlock new possibilities: lowering the accessibility barriers of text-based interfaces ( Wu et al. 2025b ) ( Zhou et al. 2025 ) , enabling hands-free interaction ( Ludwig et al. 2023 ; Jakob et al. 2025 ) , and expanding the adoption of voice conversations at a multilingual scale ( Mu et al. 2026 ) . Yet with wider deployment comes urgency around safety and fairness. Established research has surfaced stereotypes and biases in the text domain ( Nozza et al. 2022 ; Shrawgi et al. 2024 ; Mitchell et al. 2025 ) , model toxicity ( Gehman et al. 2020 ) , and failures of value alignment more broadly ( Röttger et al. 2025 ; Bu et al. 2025 ) . For speech, the stakes are compounded. The richness of audio input introduces an expanded vulnerability dimension ( Yang et al. 2024 ) . Indeed, recent studies have shown that adversaries can bypass safety mechanisms more effectively in multimodal settings ( Yang et al. 2025 ; Cheng et al. 2026 ; Pan et al. 2025 ; Chen et al. 2026b ) Despite growing attention to multimodal vulnerabilities, existing analyses of speech safety and fairness remain overwhelmingly confined to English-centric settings and synthetic voices ( Roh et al. 2025 ; Yang et al. 2025 ; Song et al. 2025 ; Li et al. 2026 ; Yu et al. 2026 ; Chen et al. 2026a ) . Synthetic voices reduce the naturalistic conditions that characterize real-world interactions, and English-only evaluations leave dangerous blind spots in safety frameworks—raising fundamental questions about the equitable distribution of AI benefits and risks across languages and communities ( Bengio et al. 2025 ; Yong et al. 2025 ; Krasnodębska et al. 2026 ) In this work, we address these limitations. (1) We begin by surveying safety and fairness reporting practices across existing speech model releases , finding that only 8% report any evaluation beyond English. (2) We then introduce RedVox , 1 1 1 Data available under gating at https://huggingface.co/datasets/FBK-MT/RedVox , see details in § Ethical Considerations . Code is available under Apache 2.0 license at: https://github.com/hlt-mt/redvox . a speech and audio benchmark for English, French, Italian, Spanish, and German. Built as a data collection through community research effort, it is—to the best of our knowledge—the first resource covering unsafe and unfair stereotypical requests in a multilingual setting built upon natural voices. (3) Finally, while red teaming and handling abusive content with human participants has been studied for text and images ( Vidgen et al. 2019 ; Quaye et al. 2024 ; Zhang et al. 2024 ) , the speech modality remains unexamined in this regard. Through a post-activity questionnaire involving the participants who contributed to RedVox , we surface its unique challenges and implications. Findings. Our review of speech models reveals that speech vulnerabilities documentation is sparse and English-centric. By evaluating eight state-of-the-art speech models on RedVox , we show that safety and fairness vulnerabilities are detectable even under naturalistic, non-challenging conditions, and that they systematically worsen in non-English languages. Besides, multimodal input—especially spoken voice—acts as a stressor beyond what text alone elicits. However, our questionnaire reveals that recording content raises distinct personal and privacy concerns (e.g., a higher sense of responsibility and fear of decontextualized identification of one’s voice with harmful content)—pointing to broader sociotechnical challenges in speech safety research that the field has yet to address. 2 On Reporting Speech Vulnerabilities We survey the state of safety and fairness assessment in the field of speech by reviewing existing model releases. We include all speech models with instruction-following capabilities that can support open-ended generation---the setting where risks are most consequential and user-facing, such as in chat assistants. 2 2 2 We exclude task-specific models, e.g., Whisper ( Radford et al. 2023 ) or Canary ( Sekoyan et al. 2025 ) . Our scope covers both SpeechLLMs, which accept speech as an input modality ( Tang et al. 2024 ) , and OmniLLMs, capable of processing arbitrary combinations of modalities—including speech—within a unified framework ( Jiang et al. 2025 ) . Where multiple model versions exist within a family, we select the most recent one. Applying these criteria, we identify 38 models—4 of which are proprietary (e.g., Gemini3)—and survey their model cards, technical reports, and accompanying papers. Following Röttger et al. 2025 , we adopt a broad notion of safety encompassing fairness, toxicity, and adversarial robustness. For each model, we annotate the languages and modalities on which it is evaluated and the evaluation methodology. The full list of annotated models is provided in Table 12 (Appendix A ). Figure 1: Safety evaluation practices across 11 speech models that provide safety documentation. We show language coverage, evaluation methodology (automatic vs. automatic+manual), and data type (public vs. private). Figure 2: RedVox Framework. ( Top ) Benchmark properties and the two request types about safety and fairness . In Request Type I ( Speech ), harmful content is vocalized and accompanied by a textual follow-up request; in Request Type II ( Audio ), harmful content appears in text only, paired with a distracting audio signal. ( Bottom ) Evaluation workflow assessing model responses on an increasing severity scale. Models Review Our review reveals gaps in safety reporting for speech models. Of the 38 models examined, only 11 document safety evaluation. 3 3 3 Three additional models address safety but do not specify the languages on which evaluations were performed. As summarised in Figure 1 , the 11 reported evaluations are overwhelmingly English-only. As exceptions, we find SeaLLMs-Audio Liu et al. 2025b and the bilingual (en/ko) Raon Speech KRAFTON 2026 . Also, the broadest coverage is found for Phi-4-Multimodal ( Abouelenin et al. 2025 ) , which evaluates across 8 languages with

Read the complete text