Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

Nithin Rao, Sasha, Nikolay, Piotr, Desh, Jagadeesh, Boris · 2026

Read Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs by Nithin Rao, Sasha, Nikolay, Piotr, Desh, Jagadeesh, Boris (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29534.txt.

Introduction

Koluguri and colleagues at NVIDIA introduce Preference-ASR, a test set for automatic speech recognition in the era of speech language models. ASR, which turns speech into text, normally gets graded after cleanup steps erase format differences. Popular test sets also follow mixed conventions for numbers, disfluencies, names, and casing. Such benchmarks cannot tell whether a model follows a user's preferred output style.

Preference-ASR instead evaluates whether systems follow natural-language style requests. Four categories are covered: normalization, entities, disfluencies, and case. The data come from seven open-source corpora through a two-stage, model-assisted pipeline checked by humans. A special normalizer skips whichever cleanup steps the active request says to keep. Benchmarking four models shows that rankings shift across request types. Differences that traditional evaluation hides become visible, and the dataset is released publicly.

Catalog record

CREATORNithin Rao, Sasha, Nikolay, Piotr, Desh, Jagadeesh, Boris
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29534

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs Nithin Rao, Sasha, Nikolay, Piotr, Desh, Jagadeesh, Boris Abstract Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions users care about. Current benchmarks therefore cannot measure whether a model follows user preferences for output style. We introduce Preference-ASR, a test set evaluating ASR systems on their ability to follow natural-language preference instructions across four categories: normalization, entities, disfluencies, and case. Built from seven open-source corpora via a two-stage LLM-assisted pipeline with human verification, it is evaluated with a preference-aware normalizer that selectively skips steps matching the active instruction. Benchmarking four models shows rankings shift across preference types, exposing quality differences traditional evaluation obscures. We publicly release the dataset 1 1 1 https://github.com/nithinraok/preference-asr-bench . Koluguri Meister Karpov Zelasko Raj Balam Ginsburg keywords automatic speech recognition, preference-aware evaluation, instruction-following, speech language models, speech LLMs † † address: 1 NVIDIA, USA † † email: [email protected] 1 Introduction Automatic speech recognition (ASR) has evolved rapidly, from hidden Markov models through end-to-end CTC and RNN-Transducer architectures to the recent class of Speech-augmented Large Language Models (SpeechLLMs) such as SALMONN [ 26 ] , Qwen-Audio [ 4 ] , and Qwen2.5-Omni [ 32 ] . Unlike earlier systems, SpeechLLMs can follow natural-language instructions that specify the desired style and format of a transcription. Yet the way we evaluate ASR has not kept pace: we still measure Word Error Rate (WER) against fixed human-annotated references, an approach that increasingly fails to capture what modern models can actually do. One root cause is that annotation conventions vary widely across popular test sets and are seldom documented in full. Number formatting alone illustrates the problem: Librispeech [ 17 ] writes numbers in spoken form and lowercases everything, SPGISpeech [ 16 ] uses inverse-normalized written form but omits proper casing, and Earnings-22 [ 5 ] mixes both forms within the same dataset. Disfluency handling is equally inconsistent: some corpora retain every filler word, repetition, and false start, while others such as SPGISpeech strip them entirely. Speaker conventions diverge too: GigaSpeech [ 3 ] contains multi-speaker audio yet provides only one speaker’s transcription, and AMI [ 12 ] mostly ignores background speech without a clear rule for which speaker is the reference. VoxPopuli [ 29 ] and Common Voice [ 2 ] add further alignment and transcription quality issues. Figure 1 illustrates two such inconsistencies: GigaSpeech writes numbers in spoken form while Earnings-22 uses written form, and AMI preserves filler words while SPGISpeech strips them. The net effect is that a model’s ranking on any single benchmark often reflects how well it happens to match that dataset’s annotation quirks, rather than its genuine transcription quality. These inconsistencies matter more now that SpeechLLMs can act on user instructions. In practice, users have specific formatting preferences: a database pipeline may need digits (Inverse Text Normalization, ITN), while a broadcast script calls for words (Text Normalization, TN). A pronunciation coaching app needs every disfluency preserved, whereas a meeting summary should omit them. Users may also want to supply domain-specific entity names (pharmaceutical terms, product names, proper nouns) to improve recognition accuracy, or request a specific casing style. No existing test set can express these preferences. Worse, standard evaluation pipelines apply blanket normalization (lowercasing, number-to-word conversion, symbol removal) that erases precisely the formatting distinctions users care about, treating them as noise rather than as meaningful signal of system capability. Normalization GigaSpeech GT: …a similar problem exists with the seventy percent rule … Earnings-22 GT: In the quarter you come up strong as thought, 10% . GigaSpeech uses spoken form; Earnings-22 uses written form Disfluencies AMI GT: Um and my findings from this is that, you know , small is beautiful. SPGISpeech GT: And my findings from this is that, small is beautiful. AMI keeps filler words; SPGISpeech strips them Figure 1: Existing ASR references exhibit inconsistent conventions across datasets: GigaSpeech writes numbers in spoken form while Earnings-22 uses written form; AMI preserves filler words while SPGISpeech strips them. Preference-ASR resolves this by conditioning each reference on an explicit user instruction. Audio + Ground Truth (7 corpora, 3,545 samples) ✓ Manual Verification GT correction Preference Classification LLM-assisted Per. category Instruction + Reference Gen. LLM-assisted Per Preference ✓ Manual Verification Human review Preference-ASR Test Set 3,210 triples (audio, instr., ref.) Stage 1 Stage 2 Figure 2: Two-stage construction pipeline for the Preference-ASR dataset. Audio samples and ground truth from seven corpora are first manually verified . Stage 1 classifies each sample into preference categories (normalization, entities, disfluencies, case) using LLM. Stage 2 generates task-specific instructions and reference texts. A final round of manual verification produces the released test set with preference-annotated triples. Prior work has touched on parts of this problem. Rich transcription [ 14 ] explored end-to-end formatted ASR output but did not tie evaluation to explicit user instructions. Multi-modal audio benchmarks such as MMAU [ 22 ] and Speech-IFEval [ 10 ] test understanding and reasoning over audio but do not target the transcription-level preferences central to ASR. Lai et al. [ 13 ] demonstrated instruction-following for speech models, though without addressing the cross-dataset inconsistencies we identify here or releasing a public test set for broader benchmarking. None of these efforts provide a test set whose ground truth is conditioned on a human-specified formatting instruction for ASR. We introduce Preference-ASR , a test set that evaluates ASR systems on their ability to follow human preference instructions. It draws from seven widely used open-source corpora, combining their strengths while explicitly controlling for the inconsistencies described above. Preference-ASR is not meant to replace existing benchmarks such as the Open ASR Leaderboard [ 8 ] ; rather, it complements them by evaluating a capability that standard test sets cannot measure: whether a model follows user instructions for ASR output formatting. Our contributions are as follows: 1. A preference-annotated English test set covering four preference categories: normalization, entities, disfluencies, and case. 2. A two-stage LLM-assisted pipeline for generating the test set, with human verification and correction. 3. A preference-aware normalizer that selectively skips normalization steps matching the active preference instruction, enabling fair WER computation across diverse formatting requirements. 4. We publicly release the test set and evaluation code to support reproducible benchmarking of SpeechLLMs on preference-following. 2 Preference-ASR Dataset 2.1 Preference Categories We organize preferences into four categories that capture the most common points of friction in real-world transcription workflows. Each category is further divided into sub-categories that can be combined in a single instruction. Normalization. Normalization deals with non-standard words whose spoken and written forms differ [ 25 ] . Choosing between Text Normalization (TN, written to spoken form) and Inverse Text Normalization (ITN, spoken to written form) directly affects downstream tasks. We define three sub-categories: • Numbers : Instructions may require digits (ITN, e.g., “22”) or spoken words (TN, e.g., “twenty-two”). • Symbols : Models may be asked to produce currency symbols and mathematical notation (ITN, e.g., “$100”, “20%”) or their verbal equivalents (TN, e.g., “one hundred dollars”, “twenty percent”). • Website Links : Instructions can specify a readable URL (ITN, e.g., “www.google.com”) or the exact sequence of letters and words spoken

Read the complete text