TidyVoice 2026 Challenge Evaluation Plan

Aref Farhadipour, Jan Marquenie, Srikanth Madikeri, Teodora Vukovic, Volker Dellwo, Kathy Reid, Francis M. Tyers, Ingo Siegert, Eleanor Chodroff · 2026

Read TidyVoice 2026 Challenge Evaluation Plan by Aref Farhadipour, Jan Marquenie, Srikanth Madikeri, Teodora Vukovic, Volker Dellwo, Kathy Reid, Francis M. Tyers, Ingo Siegert, Eleanor Chodroff (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21960.txt.

Introduction

Aref Farhadipour and a large team lay out the rules for a shared contest in speaker verification. Systems that confirm a speaker's identity degrade badly when the language changes between enrollment and testing. The field's reliance on English-centered data makes that weakness worse.

The TidyVoice 2026 Challenge, described in this evaluation plan, is designed to confront the language mismatch problem head-on. The plan spells out how systems will be tested so results can be compared fairly. Challenges like this matter because voice verification guards bank accounts and devices worldwide. And a system that fails on non-English speakers fails a large part of the world.

Catalog record

CREATORAref Farhadipour, Jan Marquenie, Srikanth Madikeri, Teodora Vukovic, Volker Dellwo, Kathy Reid, Francis M. Tyers, Ingo Siegert, Eleanor Chodroff
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TidyVoice 2026 Challenge Evaluation Plan Aref Farhadipour, Jan Marquenie, Srikanth Madikeri, Teodora Vukovic, Volker Dellwo, Kathy Reid, Francis M. Tyers, Ingo Siegert, Eleanor Chodroff Abstract The performance of speaker verification systems degrades significantly under language mismatch, a critical challenge exacerbated by the field’s reliance on English-centric data. To address this, we propose the TidyVoice Challenge for cross-lingual speaker verification. The challenge leverages the TidyVoiceX dataset from the novel TidyVoice benchmark, a large-scale, multilingual corpus derived from Mozilla Common Voice, and specifically curated to isolate the effect of language switching across approximately 40 languages. Participants will be tasked with building systems robust to this mismatch, with performance primarily evaluated using the Equal Error Rate (EER) on cross-language trials. By providing standardized data, open-source baselines, and a rigorous evaluation protocol, this challenge aims to drive research towards fairer, more inclusive, and language-independent speaker recognition technologies, directly aligning with the Interspeech 2026 theme, "Speaking Together." 0 0 footnotetext: Challenge website: https://tidyvoice2026.github.io/ 1 Introduction The TidyVoice 2026 Challenge is an evaluation for cross-lingual speaker verification, organized as part of Interspeech 2026. It targets the open problem of speaker verification under language mismatch. The main objectives are: • measure system performance under controlled cross-language conditions; • provide a simple, reproducible benchmark with clear protocols and metrics; • enable research on language-robust speaker embeddings using a public dataset. Participation is open to all teams able to comply with the rules in this plan. 2 Task Description 2.1 Task Definition The task for the TidyVoice Challenge is speaker verification: given a test segment and a target individual’s enrollment data, automatically determine whether the target individual is present in the test segment. The test segment along with the enrollment segment(s) from a designated target individual constitute a trial. The system is required to process each trial independently and output a similarity score for that trial. While the score can be in the form of a log-likelihood ratio (LLR), any similarity score that allows threshold-based decision making is acceptable. The LLR for a given trial including a test segment s is defined as \text{LLR}(s)=\log\left(\frac{P(s|H_{0})}{P(s|H_{1})}\right) (1) where P(\cdot) denotes the probability density function (pdf), and H_{0} and H_{1} represent the null (i.e., the target individual is present in s ) and alternative (i.e., the target individual is not present in s ) hypotheses, respectively. 2.2 Training Condition The training condition is defined as the amount of data/resources used to build a speaker recognition system. The task described above is evaluated over an open training condition. Open Training Condition – The open training condition removes limitations on data usage. Participants can use: • The official TidyVoiceX training partition (see Section 6 ) • Any other proprietary and/or publicly available data • Publicly available, pre-trained models (e.g., models trained on VoxCeleb [ 5 , 1 ] , VoxBlink [ 4 ] , self-supervised learning models such as wav2vec2, HuBERT, WavLM, etc.) • External, non-speech data for data augmentation (e.g., noise or reverberation from public corpora like MUSAN [ 6 ] ) However, there is a strict restriction regarding the source corpus: the only data permitted from the Mozilla Common Voice (MCV) dataset is the official TidyVoiceX training partition. The use of any other data from the MCV corpus is strictly forbidden. The official training data list will be provided on the challenge website. Participating teams must provide a sufficient description of all data resources (speech and non-speech) as well as pre-trained models used during the training and development of their systems (see Section 5.5 ). 2.3 Enrollment Conditions The enrollment condition is defined as the number of speech segments provided to create a target speaker model. In the TidyVoice Challenge, enrollment data consists of audio segments from a target speaker. The enrollment segments may vary in duration and may contain speech in one or more languages. The specific enrollment format will be provided in the trial files. 2.4 Test Conditions The test conditions in the TidyVoice Challenge are as follows: • The speech duration of the test segments will vary, typically ranging from a few seconds to approximately 60 seconds. • Trials will be conducted with test segments spoken in both the same and different languages as the enrollment segment(s). • Important: The final evaluation set uses different languages from the training and development sets. Specifically, the training and development data contain 40 languages, while the final evaluation set contains 38 unseen languages that are not present in the training and development data. This design tests the system’s ability to generalize to completely new languages. • The evaluation phase includes two trial lists: – tv26_eval-A : Enrollment utterances from seen languages (languages present in the training and development data) and test utterances from unseen languages (languages not present in the training and development data). – tv26_eval-U : Both enrollment and test utterances from unseen languages only. This list includes 38 unseen languages that are not present in the training and development data. • Information about the language of each utterance will not be disclosed during evaluation to ensure fair assessment of language-independent systems. • There will be no cross-gender trials. 3 Performance Metrics System performance will be ranked based on the following metrics, calculated from the submitted scores: 3.1 Primary Metric The primary metric for ranking will be the Equal Error Rate (EER). Evaluations will be reported on both pooled and four types of trial pair subsets to assess overall robustness and language-specific performance. In the evaluation phase, we have two different trial pair lists that all participants must evaluate and submit results for both trial lists. The four types of trial pairs in the development phase are: • Target pairs (same speaker, same language) • Target pairs (same speaker, different languages) • Non-target pairs (different speakers, same language) • Non-target pairs (different speakers, different languages) 3.2 Secondary Metric The secondary metric is the Minimum Detection Cost Function (minDCF), which provides a cost-weighted combination of false acceptance and false rejection errors. It is defined as: \text{DCF}=C_{\text{miss}}\times P_{\text{miss}}\times P_{\text{tar}}+C_{\text{fa}}\times P_{\text{fa}}\times(1-P_{\text{tar}}) (2) where C_{\text{miss}} and C_{\text{fa}} are the costs of a miss and a false alarm, respectively; P_{\text{miss}} and P_{\text{fa}} are the corresponding error probabilities; and P_{\text{tar}} is the prior probability of a target trial. The minimum value of this function over all possible decision thresholds is reported as minDCF . Evaluation Setting: We set C_{\text{miss}}=C_{\text{fa}}=1.0 and P_{\text{tar}}=0.01 . This configuration serves as our evaluation setting and is used as a secondary metric for tie-breaking and comprehensive performance analysis. 4 Rules and Requirements To ensure a fair and standardized evaluation, all participants must adhere to the following rules and requirements: • Training Data Regulations: This is an open-condition challenge, and participants are permitted to use any public or private datasets to train their systems. The use of all non-challenge data must be fully disclosed in the system description paper. However, there is a strict restriction regarding the source corpus: the only data permitted from the MCV dataset is the official TidyVoiceX training partition, as defined in Section 6 . The official training data list will be provided on the challenge website. • Prohibition of Re-identification and Data Recombination: Any attempt to determine the real-world identity of a speaker, or to link speakers in the

Read the complete text