May 2026

2026

Read May 2026 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19069.txt.

Introduction

Sajjad Abdoli and four colleagues benchmark five commercial speech-recognition systems on speech that switches between languages mid-sentence, covering Egyptian and Saudi Arabic with English, Persian with English, and German with English. Their set holds 300 samples per language pair, chosen with a two-stage filter that cut language-model scoring costs by about 91 percent. Word-error and meaning-based scores agree on system rankings, though word error inflates the gaps roughly threefold by punishing correct transliterations. ElevenLabs Scribe v2 leads on both measures, and the dataset is public.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72389
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

May 2026 Abstract Code-switching — the natural alternation between two languages within a single utterance — remains one of the most challenging and under-studied conditions for automatic speech recognition (ASR). We present a benchmark evaluating five commercial ASR providers across four language pairs: Egyptian Arabic–English, Saudi Arabic (Najdi/Hijazi)–English, Persian (Farsi)–English, and German–English, comprising 300 samples per pair selected by a two-stage pipeline combining heuristic filtering with a GPT-4o and Gemini 1.5 Pro ensemble scorer, reducing LLM costs by ≈ \approx 91%. We evaluate on both WER and BERTScore, showing that while both metrics agree on the ordinal ranking of systems for all Arabic and Persian pairs ( τ = 1.0 \tau=1.0 ), WER inflates the magnitude of quality gaps by approximately 3 × \times by penalising semantically correct transliteration choices. ElevenLabs Scribe v2 achieves the lowest WER (13.2% overall) and leads on BERTScore (0.936 overall). Difficulty-stratified analysis reveals performance gaps masked by aggregate averages, and BERT embedding projections confirm semantic proximity between reference and hypothesis despite surface-level script differences. The dataset is publicly available at https://huggingface.co/datasets/Perle-ai/ASR_Code_Switch . Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German Sajjad Abdoli * 1 1 1 Corresponding author: [email protected] , Ghassan Al-Sumaidaee * 2 2 2 ORCID: 0000-0002-5536-0252 , Clayton W. Taylor * 3 3 3 ORCID: 0009-0006-6478-8994 , Ahmad ElShiekh * 4 4 4 ORCID: 0009-0001-6837-6202 , Ahmed Rashad * * Perle AI [email protected] [email protected] [email protected] [email protected] [email protected] Keywords: automatic speech recognition, code-switching, multilingual ASR, benchmark, WER, BERTScore, Arabic, Persian, German 1. Introduction Multilingual speakers rarely confine themselves to a single language. An Egyptian software engineer on a video call might say: \arabicfont \RL أنا بحب الـ \LR style بتاعي يكون \LR simple (“I like my style to be simple”) A Persian developer describing a technical problem might say: \arabicfont \RL این \LR feature جدید کلی \LR bug داره (“This new feature has a lot of bugs”) A Saudi product manager addressing their team: \arabicfont \RL الـ \LR deadline بكره، لازم نـ \LR ship الـ \LR update (“The deadline is tomorrow, we need to ship the update”) These utterances are representative of everyday speech in multilingual workplaces across the Middle East and beyond. This phenomenon is known as code-switching (CS): the fluid, unconscious alternation between two languages within a single utterance, sentence, or even morphological unit 18 ; 1 . Despite its prevalence, code-switching remains largely absent from commercial ASR evaluation. Providers typically report WER on clean, monolingual held-out sets. This tells practitioners almost nothing about how a system will behave when deployed for genuinely multilingual populations — which today includes a substantial portion of global knowledge workers in the Middle East, South Asia, and Europe. This paper makes the following contributions: (i) A curated benchmark of 1,200 code-switching utterances (300 per language pair) across four linguistically diverse language pairs, selected by a reproducible two-stage pipeline. (ii) A comparative evaluation of five commercial ASR systems on this benchmark, reported on both WER and BERTScore. (iii) An argument and empirical demonstration that BERTScore is a more reliable primary metric than WER for Arabic and Persian code-switching due to transliteration variance. While WER and BERTScore agree on the ordinal ranking of systems ( \tau=1.0 for all Arabic and Persian pairs), WER systematically overstates the magnitude of performance differences by penalising semantically correct transliteration choices as errors. (iv) A BERT embedding space analysis providing direct visual evidence that semantically equivalent utterances are proximate regardless of script convention. 2. Related Work Code-switching has been studied extensively in linguistics 10 ; 20 , but work on ASR for code-switching lags considerably behind monolingual recognition. Early work focused on language identification preceding recognition 23 , while more recent approaches integrate end-to-end models with multilingual training data 25 ; 24 . Benchmark datasets for code-switching ASR include SEAME (Singaporean Mandarin–English, 14 , the Miami Bangor corpus 5 , and the MUCS shared task corpora 7 . More recently, 26 introduced SwitchLingua, a large-scale multilingual and multi-ethnic code-switching benchmark with synthesized text and recorded speech across 12 languages. Our work is complementary but differs in focus: rather than constructing a broad multilingual dataset, we evaluate commercial ASR systems on language pairs and regional varieties that remain underrepresented in existing benchmarks, including Persian–English and multiple Arabic–English settings spanning Egyptian and Saudi dialectal speech. Arabic–English code-switching corpora are rarer; notable exceptions include work by 11 and the ArzEn corpus 12 . Persian–English and Gulf Arabic–English code-switching datasets for ASR evaluation remain scarce. Evaluation is another recurring challenge in code-switching ASR. WER is the dominant ASR evaluation metric 15 , computing the minimum edit distance between reference and hypothesis word sequences 13 . Its limitations for code-switching include insensitivity to script-equivalent translations and no partial credit for semantically correct transcriptions in a different script convention. BERTScore 27 addresses some of these limitations by computing token-level cosine similarity in a pretrained embedding space. For multilingual evaluation, bert-base-multilingual-cased 6 maps text from 104 languages into a shared latent space, enabling meaningful comparison across scripts. Commercial ASR evaluation is also underdeveloped for code-switching. Several studies have benchmarked commercial ASR providers on specific domains or noise conditions 21 ; 2 , but systematic comparison on code-switching audio across multiple language pairs is, to our knowledge, absent from the literature. 3. Benchmark Construction 3.1. Source Datasets Our source datasets are conversational transcripts collected from real-world multilingual speakers across four language pairs: Saudi Arabic–English, Egyptian Arabic–English, Persian (Farsi)–English, and German–English. The candidate-selection counts are reported in Table 2 . Each dataset is reduced to a final benchmark split of 300 challenging utterances. Each sample consists of a single utterance, a reference transcript, and a corresponding audio filename. The final 300 samples for each language pair are selected from the Stage 2 scored pool by ranking rows according to LLM Ensemble_Score , with H_Score used as a tie-breaker. The sample-selection protocol is detailed below. 3.2. Voice Actor Recruitment and Recording The audio recordings paired with each transcript were produced by native speakers recruited through our internal contributor network. For each language pair, we sourced speakers who satisfied three criteria: (i) native or near-native fluency in the target non-English language, with the relevant regional variety (Egyptian, Najdi/Hijazi, Tehran-variety Farsi, or standard German); (ii) professional or conversational fluency in English sufficient to produce natural intra-sentential switching rather than read-aloud pronunciation; and (iii) familiarity with the technical and professional registers that dominate the corpus, since natural delivery of terms such as deployment , bottleneck , or deadline embedded in a matrix-language utterance requires the speaker to use them habitually rather than to decode them on the page. Speakers were given the reference transcripts and instructed to deliver each utterance conversationally rather than as a read performance, with latitude to adjust prosody, hesitations, and minor disfluencies to match how they would naturally produce the utterance in context. Recordings were captured in quiet indoor environments using consumer-grade headset or USB microphones, mirroring the acoustic conditions of the remote-work scenarios that the benchmark is intended to represent rather than studio-quality conditions that would overstate ASR performance in deployment. This decision was informed by prior corpus collection work where studio-grade audio was observed to materially understate the error rates that the same systems exhibited on real

Read the complete text