VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

Yi-Cheng, Yusuke, Sung-Feng, Hung-yi · 2026

Read VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech by Yi-Cheng, Yusuke, Sung-Feng, Hung-yi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17248.txt.

Introduction

Lin and three colleagues propose VIBE, a framework for evaluating generative bias in large audio models through open tasks with human speech. Existing speech fairness benchmarks rely on synthetic speech and choice questions, giving only a fragmented view of fairness. Open tasks such as personalized recommendations let stereotypical associations surface without preset options and extend easily to new tasks. Testing 12 top models reveals systematic biases in realistic scenarios: both gender and accent cues trigger statistically significant distributional shifts, and bias magnitude depends strongly on the task.

Catalog record

CREATORYi-Cheng, Yusuke, Sung-Feng, Hung-yi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23742
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech Yi-Cheng, Yusuke, Sung-Feng, Hung-yi Abstract Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing speech fairness benchmarks rely on synthetic speech and Multiple-Choice Questions (MCQs), both offering a fragmented view of fairness. We propose VIBE, a framework that evaluates generative bias through open-ended tasks such as personalized recommendations, using real-world human recordings. Unlike MCQs, our method allows stereotypical associations to manifest organically without predefined options, making it easily extensible to new tasks. Evaluating 11 state-of-the-art LALMs reveals systematic biases in realistic scenarios. We find that gender cues often trigger larger distributional shifts than accent cues, indicating that current LALMs reproduce social stereotypes. Lin Hirota Huang Lee keywords Large audio language model, Bias, Fairness, LLM † † address: 1 Graduate Institute of Communication Engineering, National Taiwan University, Taiwan 2 NVIDIA, Taiwan 3 Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan † † email: {f12942075, hungyilee}@ntu.edu.tw 1 Introduction LALMs have evolved beyond simple speech recognition [ 9 ] and classification [ 20 , 42 ] into active agents that process complex combinations of speech and text for generating open-ended text responses [ 2 ] . As these models are increasingly tasked with interpreting human intent and providing personalized recommendations, their internal biases can directly shape the social narratives presented to users. While fairness in speech technology has been studied for years, the shift toward generative modeling creates a critical evaluation gap. Most existing evaluations are designed for closed-ended tasks using performance disparity as metrics. For example, differences between demographic groups (e.g., female speakers vs. male speakers for gender bias) in Word Error Rate have been documented for speech recognition [ 35 , 25 , 22 ] . Similar metrics apply to emotion recognition [ 33 , 31 , 30 , 37 ] , intent classification [ 23 , 24 ] , and toxicity detection [ 5 ] . These works measure whether system performance differs across demographic groups, but leave a separate question unexamined: whether a model’s generated content itself reinforces social stereotypes [ 7 ] . Bias in LALMs can be triggered by either the textual content of a prompt or the acoustic characteristics of the speaker. Content-based bias closely mirrors traditional language models, where semantic gender stereotypes or linguistic prejudices are propagated through the spoken words [ 10 , 32 ] . Because these mechanisms are fundamentally similar to those in pure text models, this study focuses on speaker-triggered biases. Despite its importance, current speaker-triggered bias evaluation in LALMs often fails to mirror real-world usage. Existing benchmarks rely primarily on Rejection Rates [ 27 ] and Multiple-Choice Questions (MCQs) [ 28 , 29 ] , but both offer a fragmented view of fairness. Rejection rate measures the rigidity of a model’s safety filters rather than the fairness of its internal representations; a model that rarely refuses may still propagate harmful stereotypes [ 19 , 4 ] . Similarly, the MCQ paradigm oversimplifies the complexity of social bias by restricting the model’s expressive space to a forced-choice format [ 36 ] . Since real-world users do not provide models with predefined answer lists, evaluating free-form generation is essential to understanding a model’s true social impact [ 13 ] . To address this gap, we propose VIBE \xspace , a framework for evaluating bias in LALMs via open-ended generation. Our approach offers three key advantages. First, it evaluates models through free-form responses rather than constrained formats, allowing latent social associations to surface in the model’s natural generation space. Second, unlike MCQ-based benchmarks that require researchers to anticipate specific biases when constructing answer sets, VIBE \xspace generates unconstrained outputs that remain readily extensible to new tasks and demographic dimensions. Third, we use human-recorded audio instead of synthetic speech to better reflect real deployment conditions. Real-world recordings include diverse paralinguistic cues and phonetic variability, enabling a more robust assessment. Our evaluation of 11 LALMs spans five open-ended tasks, including story generation and personalized recommendations. The results reveal systematic biases, many of which go undetected by existing MCQ-based benchmarks. We find that bias is highly task-dependent, with narrative and recommendation prompts eliciting stronger demographic-conditioned responses than professional review settings. In particular, our results show that gender-induced cues often trigger larger distributional shifts in model output than accent cues. For example, the DeSTA model frequently assigned female speakers to service and caregiving roles like nurses or waitresses, while male speakers were associated with technical or artistic occupations such as mechanics and musicians. These findings indicate that current LALMs reproduce social stereotypes when responding to realistic vocal characteristics. Figure 1 : Overview of VIBE , the proposed generative bias evaluation framework for LALMs. 2 Methodology 2.1 Framework Overview We propose VIBE \xspace (Fig. 1 ), a generative evaluation framework that quantifies the representational biases of LALMs. Given an audio input \mathcal{X}_{audio} containing demographic cues and a task-specific prompt \mathcal{P} (e.g., ”Describe the personality of this speaker”), the target LALM M_{\theta} generates a free-form textual response \mathcal{Y}_{text} : \mathcal{Y}_{text}=M_{\theta}(\mathcal{X}_{audio},\mathcal{P}) (1) To transform the unstructured response \mathcal{Y}_{text} into quantifiable data, we employ an LLM-based extractor E_{\phi} (Qwen3-8B [ 41 ] ) [ 44 , 16 ] . The extractor maps \mathcal{Y}_{text} to a set of structured attributes \mathcal{S}=\{a_{1},a_{2},\dots,a_{n}\} , where each a_{i} represents a specific trait such as occupation, activity, or personality. The extraction prompt is tailored to each task, specifying the target attributes to identify (e.g., occupation and personality for Story , hobbies for Advisory ). This generative approach allows biases to manifest organically, reflecting its true internal associations between vocal characteristics and social stereotypes. 2.2 Evaluated Tasks To rigorously evaluate the generative bias of LALMs, we design five distinct tasks. The prompts are released on our website 1 1 1 https://anonymous.4open.science/api/repo/VIBE-52E8/file/docs/index.html . • Story Generation: The model is asked to write a short fictional story about an imaginary person for the user in the audio recording. It must explicitly define the following attributes: occupation, economic situation, educational background, field of study, family status, and personality. These attributes span social, economic, and personal dimensions, providing a broad surface for measuring demographic-conditioned variation. • Advisory: The model must suggest three specific hobbies or activities that the speaker might enjoy. This probes for gendered or cultural interests that the model may reflexively assign to certain voices. • Candidate Review: Acting as a senior HR manager, the model evaluates a candidate’s interview response. It must assess professional competency, interaction style, cultural fit, and recommended compensation. This simulates high-stakes professional bias in hiring and salary negotiation. • Hollywood Casting: The model takes the role of a Casting Director and must draft a blunt internal memo. It provides a verdict on the speaker’s star power (billing status), character function, romantic appeal, and narrative longevity (whether the character survives). This targets media-driven stereotypes and lookism. • Personal Shopper: Acting as a luxury sales associate, the model profiles a customer based on their opening line. It must predict a target department, budget level, buying triggers, and general ”vibe.” This task focuses on socioeconomic profiling and consumer stereotyping. 2.3 Data Sources & Dimensions For gender-based bias, we employ the CREMA-D [ 8 ] dataset

Read the complete text