BlasBench: An Open Benchmark for Irish Speech Recognition github.com/jyoutir/blasbench
Jyoutir Raj, John Conway · 2026
Read BlasBench: An Open Benchmark for Irish Speech Recognition github.com/jyoutir/blasbench by Jyoutir Raj, John Conway (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10736.txt.
Introduction
Raj and Conway present BlasBench, an open benchmark for Irish speech recognition that fixes a gap in multilingual testing: current benchmarks apply no Irish-aware text normalization. The harness includes a standalone normalizer preserving fadas, lenition, and eclipsis, plus reproducible scoring and per-utterance predictions. Benchmarking 12 systems across four architecture families on Common Voice and FLEURS Irish data, they find all Whisper variants exceed 100 percent word error rate through insertion-driven hallucination. Microsoft Azure leads at 22.2 percent on Common Voice, while the best open model reaches 30.65 percent.
Catalog record
| CREATOR | Jyoutir Raj, John Conway |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23828 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BlasBench: An Open Benchmark for Irish Speech Recognition github.com/jyoutir/blasbench Jyoutir Raj, John Conway Abstract Existing multilingual benchmarks include Irish among dozens of languages but apply no Irish-aware text normalisation, leaving reliable and reproducible ASR comparison impossible. We introduce BlasBench , an open evaluation harness that provides a standalone Irish-aware normaliser preserving fadas, lenition, and eclipsis; a reproducible scoring harness and per-utterance predictions released for all evaluated runs. We pilot this by benchmarking 12 systems across four architecture families on Common Voice ga-IE and FLEURS ga-IE. All Whisper variants exceed 100% WER through insertion-driven hallucination. Microsoft Azure reaches 22.2% WER on Common Voice and 57.5% on FLEURS; the best open model, Omnilingual ASR 7B, reaches 30.65% and 39.09% respectively. Models fine-tuned on Common Voice degrade 33–43 points moving to FLEURS, while massively multilingual models degrade only 7–10—a generalisation gap that single-dataset evaluation misses. 1 Introduction State-of-the-art multilingual systems fail on out-of-distribution audio in Irish. Microsoft Azure reaches 22.2% WER on read speech (Common Voice ga-IE) and 57.5% WER on the more naturalistic FLEURS ga-IE. Every Whisper variant we evaluate ( Radford et al. 2023 ) produces more tokens than the reference contains, exceeding 100% WER on both datasets (Tables 1 , 2 ). Irish presents a genuine difficulty: initial consonant mutations, eclipsis, and a spelling system that encodes grammatical contrasts mean that language-agnostic normalisation silently discards linguistically meaningful variation. The problem is not that nobody has worked on Irish ASR. The ABAIR group at Trinity College Dublin has produced a sustained research programme culminating in their Fotheidil system at 10.9% WER ( Lonergan et al. 2025 ) . Qian et al. 2024 fine-tune Whisper on Irish. Community wav2vec2 fine-tunes exist on HuggingFace ( Faste 2022 ) . The IWSLT shared task has produced Irish–English speech translation systems ( Ahmad et al. 2024 ; Moslem 2024 ) . Multilingual benchmarks—FLEURS ( Conneau et al. 2022a ) , MMS ( Pratap et al. 2023 ) , ML-SUPERB ( Shi et al. 2023 ) —include Irish among dozens or hundreds of languages. These efforts span architecture families and data regimes, yet they have not been placed on a common footing. The gap is methodological rather than empirical. ABAIR’s best result is on private MíleGlór data; their published Common Voice and FLEURS-R figures cover only their own systems. Community models self-report on different Common Voice versions with different normalisers. Multilingual benchmarks include Irish without Irish-specific normalisation or targeted analysis. 1 1 1 ML-SUPERB and XTREME-S ( Conneau et al. 2022b ) include Irish data points but do not compare end-user ASR systems on Irish specifically. We did not identify a prior open, Irish-specific comparison of end-user ASR systems across architecture families under a shared Irish-aware evaluation protocol. We present BlasBench, 2 2 2 Harness: github.com/jyoutir/blasbench . Per-utterance predictions and aggregate results for every run in this paper are released as v0.1.0 . an open benchmark and evaluation harness for Irish ASR. The harness applies Irish-aware text normalisation before scoring, so that orthographic variants reflecting mutation and dialect do not inflate error rates artificially. A 12-model comparison across commercial, large-scale multilingual, and fine-tuned systems constitutes the first use of the benchmark; reproducible, language-aware scoring is also a prerequisite for any automated research pipeline that requires a stable signal. Dedicated low-resource benchmarks for Urdu ( Arif et al. 2025 ) and Scottish Gaelic ( Klejch et al. 2025 ) have made the same case for their respective languages; BlasBench brings it to Irish. 2 The BlasBench Benchmark 2.1 Task and scope BlasBench evaluates Irish speech-to-text systems. Given Irish audio, a system produces Irish text; the benchmark scores the output against a reference transcription using WER and CER, computed with Irish-aware normalisation. 2.2 Datasets Two public datasets, chosen to measure both in-distribution and out-of-distribution performance: • Common Voice 25.0 ga-IE ( Ardila et al. 2020 ) : 874 community-recorded read-speech test utterances in the released BlasBench evaluation slice; the utterance count is recorded in the released run metadata. Most wav2vec2 fine-tunes we evaluate were trained on earlier CV versions. • FLEURS ga-IE ( Conneau et al. 2022a ) : 842 test utterances in the released BlasBench evaluation slice; the utterance count is recorded in the released run metadata. The speech is professionally recorded native-speaker read speech. Based on public model cards, no model in our benchmark reports FLEURS in its training mix. The released prediction files and run metadata make each evaluation slice auditable. Using two datasets with different recording conditions is central to the benchmark design: it exposes generalisation failures invisible to single-dataset evaluation (§ 5.2 ). 2.3 Evaluation pipeline Audio input (16kHz) Model wrapper (any architecture) Irish-aware normaliser Global WER / CER Bootstrap 95% CI raw text Figure 1: BlasBench evaluation pipeline. The Irish normaliser (highlighted) is the language-specific component. The pipeline (Figure 1 ) proceeds in four stages: 1. Inference. Audio is passed through a model wrapper that returns raw text. The wrapper API is standardised: any system that accepts 16kHz audio and returns a string can be evaluated. 2. Normalisation. Both reference and hypothesis are normalised using an Irish-aware pipeline (§ 2.4 ). 3. Scoring. WER and CER are computed via global aggregation: total substitutions, insertions, and deletions across all utterances divided by total reference units. This corpus-level approach avoids the bias of per-sentence averaging and matches the convention used in low-resource benchmarking work ( Arif et al. 2025 ) . 4. Confidence intervals. Bootstrap 95% CIs ( Bisani and Ney 2004 ) are computed with 1,000 resamples at a fixed seed (42), resampling utterance-level error counts and recomputing the global aggregate. CIs for every reported point estimate are released alongside the predictions in each run’s results.json ; tables show point estimates for compactness. 2.4 Irish-aware normalisation Standard ASR normalisers lowercase text, strip punctuation, and collapse whitespace. Recent work on multilingual ASR evaluation shows that language-agnostic normalisation can materially distort reported error rates ( Manohar et al. 2024 ) . For Irish, it destroys linguistically significant information. Irish uses acute accents (fadas) on five vowels: á, é, í, ó, ú. These are phonemically contrastive: fear “man” vs féar “grass.” Standard NFKD normalisation decomposes accented characters and may strip the combining accent. Our normaliser applies NFC first, guaranteeing fadas survive. Irish also marks grammar through initial mutations. Lenition: bean “woman” \to bhean “his wife.” Eclipsis: bean \to mbean “of women.” Our normaliser preserves these. ABAIR’s pipeline includes text-processing components used inside Fotheidil ( Lonergan et al. 2025 ) but no Irish ASR normaliser has been released standalone. To our knowledge, BlasBench ships the first standalone open-source Irish ASR evaluation normaliser released specifically for Irish ASR scoring; unreleased internal tools may exist. 2.5 Released artifacts For each (model, dataset) run, BlasBench produces: • Per-utterance predictions with sample ID, reference, hypothesis, WER, CER • Aggregate results with S/I/D breakdown and bootstrap CIs • Run metadata recording dataset name/split/count, model identity, evaluation configuration, and software versions The evaluation harness and released predictions are available at the repository linked under the title. Future systems can be compared against these predictions without re-running prior models. 2.6 Reproducibility and extensibility Adding a new model requires implementing a single function: given 16kHz audio, return a string. The harness handles normalisation, scoring, CI