Fair-Gate: Fairness-Aware Interpretable Risk Gating for Sex-Fair Voice Biometrics

Yangyang Qu, Massimiliano Todisco, Chiara Galdi, Nicholas Evans · 2026

Read Fair-Gate: Fairness-Aware Interpretable Risk Gating for Sex-Fair Voice Biometrics by Yangyang Qu, Massimiliano Todisco, Chiara Galdi, Nicholas Evans (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11360.txt.

Introduction

Yangyang Qu and coauthors tackle sex-related gaps in voice biometrics, where verification accuracy can still hide unequal treatment. They trace the gaps to shortcut learning, which exploits links between sex and identity during training, and to feature entanglement, where sex-linked acoustic variation overlaps the cues of identity. Their Fair-Gate pipeline counters both. It uses risk extrapolation across proxy sex groups plus a complementary gate. The gate routes features into identity and sex branches and shows its routing mask. This improves the fairness-utility balance on VoxCeleb1 tests.

Catalog record

CREATORYangyang Qu, Massimiliano Todisco, Chiara Galdi, Nicholas Evans
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27761
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11360

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fair-Gate: Fairness-Aware Interpretable Risk Gating for Sex-Fair Voice Biometrics Yangyang Qu, Massimiliano Todisco, Chiara Galdi, Nicholas Evans Abstract Voice biometric systems can exhibit sex-related performance gaps even when overall verification accuracy is strong. We attribute these gaps to two practical mechanisms: (i) demographic shortcut learning, where speaker classification training exploits spurious correlations between sex and speaker identity, and (ii) feature entanglement, where sex-linked acoustic variation overlaps with identity cues and cannot be removed without degrading speaker discrimination. We propose Fair-Gate , a fairness-aware and interpretable risk-gating framework that addresses both mechanisms in a single pipeline. Fair-Gate applies risk extrapolation to reduce variation in speaker-classification risk across proxy sex groups, and introduces a local complementary gate that routes intermediate features into an identity branch and a sex branch. The gate provides interpretability by producing an explicit routing mask that can be inspected to understand which features are allocated to identity versus sex-related pathways. Experiments on VoxCeleb1 show that Fair-Gate improves the utility–fairness trade-off, yielding more sex-fair ASV performance under challenging evaluation conditions. I Introduction Recent progress in deep speaker embedding architectures has led to substantial gains in automatic speaker verification (ASV), a core technology for voice biometrics. Despite these improvements, ASV systems can still exhibit systematic performance differences across demographic groups, especially sex. Although subgroup performance may appear similar when each group is evaluated at its own operating point, such reporting can mask disparities that emerge when deploying a single global decision threshold shared by all users, which is the common practical setting [ 29 , 13 ] . Recent studies have documented these disparities and examined how dataset composition, demographic imbalance, and evaluation design affect fairness, commonly defined as comparable verification error rates across groups under a shared operating point [ 8 , 2 , 10 , 7 , 29 , 13 , 3 , 12 ] . Importantly, these effects persist under standard protocols and operating points, suggesting systematic group-dependent model behavior rather than evaluation artifacts [ 29 , 13 ] . A common mitigation strategy is to reduce the extent to which ASV embeddings encode sensitive attributes such as sex. This is often implemented through adversarial objectives or auxiliary prediction losses, sometimes combined with reweighting or group-aware fusion [ 9 , 1 , 23 , 14 , 26 , 4 ] . However, in ASV, sex-linked acoustic cues are not purely nuisance factors: pitch, timbre, and resonance correlate with sex but also carry identity-relevant information. Consequently, enforcing strong sex invariance can suppress useful speaker cues and degrade verification performance [ 23 ] . Related trade-offs have also been observed in privacy-oriented speaker anonymization and sex obfuscation, where reducing sex information can harm downstream utility [ 27 , 28 , 25 ] . This motivates a different goal: rather than enforcing global sex invariance, we seek to control where sex-linked variation is represented and how strongly it perturbs verification behavior under a shared decision threshold. Inspired by causal fairness formulations [ 16 ] , we distinguish (i) the causal effect of sex on speech acoustics (e.g., F_{0} and formants) from (ii) dataset-induced correlations between sex and speaker identity in the training data. As illustrated in Fig. 1 , standard discriminative training can exploit such correlations as demographic shortcuts, shifting score distributions differently across groups, and producing unequal subgroup error rates under a shared threshold. Fig. 1: Desired decision vs. demographic shortcut in speaker verification under a shared threshold. A verifier should base its decision on identity evidence by comparing enrollment and test utterances (solid arrow). However, because sex affects acoustics (e.g., F_{0} and formant structure) and can be spuriously correlated with speaker identity in the training data, the model may also exploit sex-linked cues as a shortcut (dashed arrow). Such shortcut reliance can shift score distributions differently for male and female speakers, leading to subgroup error-rate gaps when deploying a single global decision threshold. Fig. 2: Overview of Fair-Gate. The encoder produces frame-level features \mathbf{U} , which are complementarily soft-routed by a local mask A (gate) into an identity branch and a sex branch. The identity branch produces the embedding z_{\mathrm{id}} , which is the only representation used for automatic speaker verification (ASV) at inference. During training, the sex branch learns a sex embedding z_{\mathrm{sex}} and predicts proxy sex labels \hat{s} via a sex classifier ( \mathcal{L}_{\mathrm{sex}} ). The identity branch is optimized for speaker classification ( \mathcal{L}_{\mathrm{spk}} ), regularized by Risk Extrapolation (REx) across proxy sex groups ( \mathcal{L}_{\mathrm{rex}} ), and constrained by an adversarial sex classifier implemented through a Gradient Reversal Layer (GRL) ( \mathcal{L}_{\mathrm{adv}} ). A decorrelation loss ( \mathcal{L}_{\mathrm{decor}} ) encourages separation between z_{\mathrm{id}} and z_{\mathrm{sex}} , while gate regularizers ( \mathcal{L}_{\mathrm{cap}} , \mathcal{L}_{\mathrm{sat}} ) prevent degenerate routing. This analysis highlights two fairness-relevant failure modes: (i) shortcut learning , where speaker classification relies on sex-conditioned correlations that do not generalize equally across groups; (ii) feature entanglement , where sex-related variation mixes with identity cues in the embedding used at inference, making it difficult to reduce subgroup error gaps without sacrificing verification performance. Motivated by these mechanisms, we propose Fair-Gate , a unified training framework that improves the utility–fairness trade-off by addressing shortcut learning and feature entanglement in a single pipeline. In the following, we use the term utility to refer specifically to verification performance, in order to clearly distinguish it from fairness performance. First, we apply Risk Extrapolation (REx) [ 15 ] across proxy sex groups to penalize differences in speaker-classification risk between groups, discouraging group-specific shortcuts. Second, we introduce a complementary local soft-routing gate that partitions intermediate features into two additive components routed to an identity branch and a sex branch, while preserving the original feature dimensionality. The sex branch provides an explicit pathway for sex-linked variation during training, reducing its leakage into the identity embedding used for verification at inference. Our contributions are: • We provide a causal analysis of sex-related bias in ASV, separating inherent sex-linked acoustic variation from dataset-induced correlations. • We propose Fair-Gate, combining Risk Extrapolation (REx) across proxy sex groups with a complementary local gating mechanism and branch-specific objectives to limit sex leakage into the deployed embedding. • We demonstrate improved utility–fairness trade-offs on VoxCeleb, and provide ablations that clarify the roles of complementary routing and branch-specific training objectives. Despite sex and gender terms being commonly used interchangeably, there is a generally, though not universally accepted distinction. We adopt the terminology in [ 21 ] and expressly avoid references to gender which refers to socially constructed roles and behaviour. Instead, we refer only to sex, which refers to biological attributes [ 24 ] . II Fair-Gate Framework Fair-Gate extends a standard ECAPA-style speaker verification pipeline with complementary feature routing and fairness-aware training objectives. The framework illustrated in Fig. 2 consists of three key components: (i) a shared encoder that extracts frame-level representations, (ii) a local complementary gate that routes these representations into an identity branch and a sex branch, and (iii) branch-specific objectives, including risk variance equalization, that promote speaker discrimination while reducing sex-dependent disparities under a shared operating threshold. At inference, only the identity branch is retained for verification. Given an enrollment–test pair (x_{a},x_{b}) , the verification score is computed using cosine similarity between identity embeddings: \mathrm{score}(x_{a},x_{b})=\cos\!\big(z_{\mathrm{id}}(x_{a}),z_{\mathrm{id}}(x_{b})\big). (1)

Read the complete text