SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR
Kavya, Arghya, Kush, Kumarmanas · 2026
Read SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR by Kavya, Arghya, Kush, Kumarmanas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20712.txt.
Introduction
Kavya Manohar and three colleagues present SCRIBE, a diagnostic framework for speech recognition in Indic languages. Word error rate collapses distinct mistakes into one number and penalizes agglutinative languages, where valid word merges inflate scores. SCRIBE instead splits errors into lexical, punctuation, numeral, and domain-entity rates, using sandhi-tolerant alignment with domain vocabulary. Human checks confirm it matches expert judgment where word error rate does not, and rich transcription models are released for Hindi, Malayalam, and Kannada.
Catalog record
| CREATOR | Kavya, Arghya, Kush, Kumarmanas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23806 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20712
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR Kavya, Arghya, Kush, Kumarmanas Abstract Automatic speech recognition replaces typing only when correction costs less than manual entry, a threshold determined by error types, not counts: fixing a misrecognized domain term costs far more than inserting a comma. Word error rate (WER) fails on two fronts: it collapses distinct error categories into a single scalar, and it structurally penalizes agglutinative languages where valid sandhi merges inflate scores. We introduce SCRIBE, a diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi -tolerant alignment with domain vocabulary injection. Human validation confirms SCRIBE aligns with expert judgment where WER does not. We release SCRIBE, an LLM curation pipeline, benchmarks, and open-weight rich transcription models for Hindi, Malayalam, and Kannada. Manohar Bhattacharya Juvekar Nethil [Script=Devanagari] [Script=Malayalam, Scale=0.9] [Script=Devanagari] [Script=Malayalam, Scale=0.9] keywords ASR Evaluation, Rich Transcription, Morphological Alignment, Indic Languages, Diagnostic Metrics † † address: Adalat AI, India † † email: {kavya,arghya,kush,manas}@adalat.ai \fontspec_if_language:nT ENG \addfontfeature Language=English1 Introduction \fontspec_if_language:nT ENG \addfontfeature Language=English Verbatim Corpora \fontspec_if_language:nT ENG \addfontfeature Language=English LLM Curation Pipeline Formatting & Domain Injection Release 1: Pipeline \fontspec_if_language:nT ENG \addfontfeature Language=English ASR Model Training (Hindi, ML, KN) \fontspec_if_language:nT ENG \addfontfeature Language=English SCRIBE Framework Diagnostic Evaluation Release 2: Framework \fontspec_if_language:nT ENG \addfontfeature Language=English Final Weights & Benchmarks Release 3 \fontspec_if_language:nT ENG \addfontfeature Language=English Domain Data (Optional) Diagnostic Feedback Loop with Categorical Analysis \fontspec_if_language:nT ENG \addfontfeature Language=English , whichdecomposesfailuresintolexical , punctuation , numeral , anddomain - specificentityerrorrates , respectively . Byutilizingsandhi - tolerantalignmentandcategoricaldecomposition , SCRIBEreplacesthemonolithicWERwithanactionabledevelopmentsignal , enablingtargetedremediationforrichtranscriptiontasks . Tosummarize , ourmajorcontributionsinthispaperare : \fontspec_if_language:nT ENG \addfontfeature Language=English• SCRIBE , released as an open-source evaluation tool, providing sandhi -tolerant alignment and categorical error decomposition, proposed as a replacement for monolithic WER wherever ASR serves as a scribe. \fontspec_if_language:nT ENG \addfontfeature Language=English• A structured annotation schema and validation procedure for categorical ASR metrics, with dimension-specific scales rated independently by expert linguists, demonstrating that SCRIBE aligns with human judgment where WER does not. \fontspec_if_language:nT ENG \addfontfeature Language=English• A reproducible recipe for Indic rich transcription : an LLM-based data curation pipeline, two new benchmarks (FLEURS-RO for general and IN22-Legal for domain evaluation), and the first open-weight rich transcription models for Hindi, Malayalam, and Kannada. \fontspec_if_language:nT ENG \addfontfeature Language=English2 Related Work Rich Transcription Models: While models like Whisper [ 21 ] and Canary [ 22 ] demonstrate the feasibility of joint acoustic-orthographic modeling, the open-source Indic ecosystem remains dominated by verbatim-only models [ 3 , 6 , 5 ] . Current pipelines for formatted output often rely on decoupled inverse text normalization [ 20 ] , which ignores prosodic cues and homophone resolution. SCRIBE bridges this by providing a recipe for native rich transcription in Indic ASR. Evaluation Limitations: While character error rate (CER) sidesteps word-boundary issues in agglutinative languages [ 14 ] , it and WER lack diagnostic signals. CER is semantically blind, weighting functional suffix changes identically to root morpheme substitutions. Categorical frameworks like Beyond Levenshtein [ 16 ] move toward nuanced evaluation but rely on normalization that destructively strips lexically indispensable Indic vowel signs ( matras ) and diacritics [ 17 ] . Similarly, 1:1 word alignment—shared by word information lost (WIL) and word information preserved (WIP) [ 18 ] —cannot resolve valid sandhi (word-boundary merges) common in Dravidian languages [ 17 ] . Semantic metrics like Semascore [ 15 ] prioritize global meaning but remain blind to fatal numeral or negation failures that render professional dictation unusable. While the Orthographically-Informed WER leverages LLMs to capture permissible variations [ 4 ] , the approach is computationally expensive and fails to resolve the structural alignment shifts caused by sandhi . SCRIBE addresses these gaps through diacritic-preserving normalization, deterministic sandhi -tolerant alignment, and categorical error decomposition. \fontspec_if_language:nT ENG \addfontfeature Language=English3 The SCRIBE Framework SCRIBE is organized into three phases: tokenization and domain shielding, a sandhi-aware alignment engine, and categorical error aggregation. The framework outputs a diagnostic error vector \mathbf{E} where each component maps to a specific remediation strategy. \fontspec_if_language:nT ENG \addfontfeature Language=English3.1 Phase 1: Tokenization and Domain Shielding The framework transforms reference R and hypothesis H into typed tokens (w_{i},t_{i}) , where t_{i}\in\{\text{lexeme, numeral, punctuation, domain-entity}\} . Unlike standard tokenizers that strip or blindly isolate punctuation, in SCRIBE standard punctuation and Indic-specific marks (e.g., the Hindi danda ) become independent tokens, while punctuation within numerals and compound words (e.g., \fontspec_if_language:nT ENG \addfontfeature Language=English22.05.2023 , \fontspec_if_language:nT ENG \addfontfeature Language=Englishice-cream ) are preserved to maintain lexical integrity. User-defined domain entities are injected via a regex-based shielding layer to treat them as atomic units, preventing spurious fragmentation. \fontspec_if_language:nT ENG \addfontfeature Language=English3.2 Phase 2: Sandhi-Aware Alignment Engine An alignment is a pairing of reference, R and hypothesis, H positions that accounts for insertions, deletions, standard 1:1 substitutions, and Sandhi -motivated 1:2 (split) and 2:1 (merge) mappings. We seek the alignment maximizing a total score dp[i][j] , calculated via extended dynamic programming in Equation \fontspec_if_language:nTENG\addfontfeatureLanguage=English1 . dp[i][j]=\max\left\{\begin{aligned} &dp[i\text{-}1][j\text{-}1]+S(r_{i},h_{j})&&\text{(match/sub)}\\ &dp[i\text{-}1][j]+\gamma(t^{R}_{i})&&\text{(deletion)}\\ &dp[i][j\text{-}1]+\gamma(t^{H}_{j})&&\text{(insertion)}\\ &dp[i\text{-}1][j\text{-}2]+\Sigma_{\text{split}}&&\text{(Sandhi-split)}\\ &dp[i\text{-}2][j\text{-}1]+\Sigma_{\text{merge}}&&\text{(Sandhi-merge)}\end{aligned}\right. \fontspec_if_language:nT ENG \addfontfeature Language=English(1) The scoring function S(r_{i},h_{j}) anchors the alignment on exact matches ( \alpha=+4.0 ) while buffering against the acoustic near-misses common in Indic scripts (e.g., deva \addfontfeature Script=Devanagari \fontspec_if_language:nT HIN \addfontfeature Language=Hindiखाना: khana vs deva \addfontfeature Script=Devanagari \fontspec_if_language:nT HIN \addfontfeature Language=Hindiगाना: gana ). To prevent alignment drift, a category-clash penalty \beta=-3.0 is applied if t^{R}_{i}\neq t^{H}_{j} . For same-category substitutions, we employ a Levenshtein-buffered penalty \delta=-1.5-(0.2\cdot d) , where d is the character distance between r_{i} and h_{j} . Sensitivity analysis on our target languages confirms that a Unicode-level distance of d\leq 2 optimally captures minor orthographic variations—such as matra shifts or gemination—without triggering the cascading deletion-insertion pairs typical of standard WER evaluation. Sandhi scores, \Sigma , resolve 1:2 or 2:1 mappings by validating phonetic plausibility. A transition is valid if the fused form s matches the prefix of w_{1} and suffix of w_{2} . We score this as \Sigma=\alpha+\sigma-d(b_{split},b_{mid})/|s| , where \sigma=-0.5 is the sandhi penalty. If the boundary distance d>2 , the transition is invalidated per Indic morphophonological rules [ 2 , 9 , 23 ] . Figure illustrates SCRIBE’s ability to correctly resolve these complex word merges like ‘ innu allenkil \rightarrow innallenkil ’ and splits like ‘ naleyakatte \rightarrow nale akatte ’ in Malayalam. SCRIBE Sandhi-Aware Alignment mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamഇന്ന് mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamഅല്ലെങ്കിൽ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamനാളെയാകട്ടെ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamഇന്നല്ലെങ്കിൽ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamനാളെ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamആകട്ടെ MERGE (2:1) SPLIT (1:2) E_{lex}=0\% : Correct resolution of linguistic merges and splits Standard 1:1 Alignment (JIWER) mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamഇന്ന് mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamഅല്ലെങ്കിൽ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamനാളെയാകട്ടെ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamഇന്നല്ലെങ്കിൽ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamനാളെ mlym \addfontfeature Script=Malayalam \fontspec_if_language:nT MAL \addfontfeature Language=Malayalamആകട്ടെ sub sub sub WER=100\% : Alignment shift due to word-splits and merges \fontspec_if_language:nT ENG \addfontfeature Language=English . Weemployacombineddenominator N_ comb = ∑_t ∈ T total [t] tocalculatecategoricalrates: ER_t = (sub[t] + ins[t] + del[t]) / N_ comb .Thisunifiedscalingpreventsmisleadinglyhighratesfromisolatedfailuresofsparsecategories.Toaccountforvalidformattingchoicesinprofessionaldictation,SCRIBEoptionallynormalizesdateandnumeraldelimiters,ensuringthatacceptableorthographicvariationsdonotinflate ER_num .TheframeworkgeneratesdetailedreportswithabsoluteerrorcountstofacilitategranulardiagnosticvisualizationandthedevelopmentoftargetedremediationstrategiesforASRsystems.\par \fontspec_if_language:nT ENG \addfontfeature Language=English4 Experimental Setup We validate SCRIBE through a complete experimental cycle of rich transcription model development for Hindi, Malayalam, and