Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild

Berkin, Chen, Eduardo, Arda, Atila · 2026

Read Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild by Berkin, Chen, Eduardo, Arda, Atila (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07354.txt.

Introduction

Durmus and four colleagues introduce Contextual Earnings-22, a speech-recognition benchmark built around rare, context-defined vocabulary in real earnings calls. Academic benchmarks have plateaued because they lean on common words, while industrial use hinges on custom terms that shape transcript usefulness. The open dataset adds realistic custom-vocabulary contexts to Earnings-22. Six strong baselines for keyword prompting and keyword boosting reach comparable, much-improved accuracy when scaled from prototypes to large systems.

Catalog record

CREATORBerkin, Chen, Eduardo, Arda, Atila
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24172
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07354

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild Berkin, Chen, Eduardo, Arda, Atila Abstract The accuracy frontier of speech-to-text systems has plateaued on academic benchmarks. 1 1 1 The top-11 models on Hugging Face OpenASR Leaderboard [ 25 ] as of December 2025 achieve very close average WER across 8 commonly used benchmark datasets. Part of this narrow gap is also attributed to verbatim vs non-verbatim behavior variance across models and may point to a near-saturation of WER improvements. In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise. We hypothesize that the primary difference between the two is contextual conditioning: Academic benchmarks are dominated by frequently encountered general vocabulary that is relatively easy to recognize compared with rare and context-defined custom vocabulary that has disproportionate impact on the usability of speech transcripts. Despite progress on contextual speech-to-text, there is no standardized benchmark. We introduce Contextual Earnings-22, an open dataset built upon Earnings-22, with realistic custom vocabulary contexts to foster research and reveal latent progress. We set six strong baselines for two dominant approaches: keyword prompting and keyword boosting. Experiments show both reach comparable and significantly improved accuracy when scaled from proof-of-concept to large-scale systems. 1 1 1 The top-11 models on Hugging Face OpenASR Leaderboard [ 25 ] as of December 2025 achieve very close average WER across 8 commonly used benchmark datasets. Part of this narrow gap is also attributed to verbatim vs non-verbatim behavior variance across models and may point to a near-saturation of WER improvements. Durmus Cen Pacheco Okan Orhon keywords speech-to-text, speech recognition, context biasing, custom vocabulary, keyword recognition † † address: 1 Argmax, Inc., USA 2 University of California, Los Angeles, USA † † email: {berkin,chen,eduardo,arda,a}@argmaxinc.com 1 Introduction and Related Work Speech-to-text (STT) has reached high levels of accuracy on widely used academic benchmarks, to the point that reported word error rate (WER) improvements are often marginal across top-performing systems. 1 1 footnotemark: 1 This apparent mismatch suggests that commonly reported benchmark WER may no longer be a sufficient proxy for real-world transcript utility. A key driver of this mismatch is contextual conditioning : in many applications, a small set of context-defined (custom) terms disproportionately determines whether a transcript is usable. In earnings calls, for example, a transcript can be otherwise fluent yet fail in practice if it repeatedly misrecognizes company, product, or person names. This creates a regime where overall WER can be near-saturated while custom term accuracy remains far from solved. This paper studies the general problem of speech recognition with custom vocabulary (also called context biasing in prior work [ 3 , 2 , 11 , 26 , 15 , 10 , 13 ] ). We use custom vocabulary to emphasize the problem setting rather than a specific method. Figure 1: Keyword F-Score vs Word Error Rate comparison across different systems and keyword contexts. Contextual STT methods. In practice, two methods dominate deployments and the literature. Keyword boosting integrates a term list into decoding to increase their likelihood when acoustically plausible, spanning shallow-fusion style approaches and GPU-accelerated decoders [ 3 , 2 , 11 , 26 , 15 , 10 ] . Keyword prompting conditions STT on a keyword list via a textual prompt [ 22 , 7 , 21 ] . Recent work also studies scalability to long bias lists and mitigations such as ranking/selection of bias terms [ 13 ] . Benchmarks and datasets for contextual STT. Evaluation for contextual STT remains fragmented. Many influential contextual-ASR papers rely on private or synthetic evaluation setups, often constructed by injecting “rare words” from a general-domain corpus and adding random distractors. For example, [ 15 ] evaluates primarily on LibriSpeech with per-utterance synthetic bias lists (rare reference words plus up to thousands of distractors) and additionally reports results on in-house data; the accompanying “IS21 deep bias” recipe is publicly released, but the synthetic construction does not reflect naturally occurring, domain-specific entity inventories. Similarly, streaming and transducer-based contextualization work such as [ 26 ] evaluates on LibriSpeech and internal voice-assistant data, limiting cross-paper comparability. More recent scaling work [ 13 ] studies large bias lists using LibriSpeech with bias words derived from named entities and the IS21 bias list, again yielding an ad-hoc but controlled setup rather than a domain-realistic benchmark. Earnings-22 [ 8 ] corpora provide a natural public domain where proper nouns are dense and errors on them are high-impact. ConEC [ 14 ] augments Earnings-22 with biasing lists derived largely from external sources (e.g., slides, earnings releases, and participant metadata), alongside transcript cleanup and sentence-level segmentation, and reports WER-based evaluation on long-form audio. Separately, Earnings22-Cleaned-AA [ 4 , 1 ] targets reference quality by cleaning transcripts for a very small subset of Earnings-22, but does not introduce contextual vocabularies or contextual evaluation protocols. Despite this progress, existing public resources still lack a widely adopted, standardized benchmark that pairs context-dense short clips with direct (in-conversation) custom-vocabulary contexts and evaluates both an idealized precise-context regime and a deployment-realistic distractor regime, enabling apples-to-apples comparison across prompting vs. boosting. Contextual Earnings-22. 2 2 2 Code and dataset will be released upon acceptance. We introduce Contextual Earnings-22, a public dataset built on Earnings-22 [ 8 ] that targets the most consequential contextual errors in earnings-call transcription: person, company, and product names . Each audio segment is paired with realistic custom-vocabulary contexts, and we evaluate two practical scenarios: local context without distractors and global context with realistic distractors. To support reliable comparison, we manually review and correct the transcripts where needed, addressing earnings-call transcript artifacts at substantially broader coverage than prior cleaned subsets [ 4 , 1 ] . We build on a previously released open-source evaluation harness 2 2 2 Code and dataset will be released upon acceptance. by adding keyword boosting evaluation protocols and reproducible baselines. We set strong baselines spanning the two dominant contextualization families: keyword prompting via widely used STT APIs [ 22 , 7 , 21 ] and keyword boosting via a scalable CTC-WS pipeline [ 3 ] . 2 Methodology Earnings-22 Transcript LLM (GPT-5) Contextual Keywords Clip transcript around each keyword Transcript Segments with Contextual Keywords Force align with wav2vec Earnings-22 Audio File ( \sim 1 hour) Clip audio around each keyword Manually review and correct 15-second Audio Clips with Contextual Keywords Figure 2: Contextual Earnings-22 creation pipeline. Manual review substantially reduced transcript artifacts in the overlapping portion of the dataset: 98.7% of the samples are free of inaudible and <unk> tags, and 29.5% of clips receive word-level corrections, including spelling fixes as well as word insertions and deletions, affecting 411 words in total. 2.1 Contextual keyword extraction Using Earnings-22 [ 8 ] audio–transcript pairs ( \sim 1h per call), we curate samples with contextual keywords focusing on three categories: person , company , and product names . For each Earnings-22 sample, we run an LLM-based named-entity pass (GPT-5) over the transcript to obtain candidate keywords. To make the resulting keyword lists stable and evaluation-friendly, we apply deterministic post-processing: (i) de-duplication across surface forms, (ii) punctuation and whitespace normalization, and (iii) filtering of generic strings. The final per-call keyword inventory defines that call’s global context list, which naturally varies across calls and includes realistic distractors

Read the complete text