SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation

Priyaranjan Pattnayak · 2026

Read SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation by Priyaranjan Pattnayak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02548.txt.

Description

Pattnayak proposes a fairer score for speech recognition in multi-script settings. Word Error Rate can overstate errors. The same words may be written in different scripts. Romanized output from a model is a common case. His method, SN-WER, fixes the comparison first. Reference and hypothesis are transliterated into one canonical script per language. The standard score is then computed.

Tests span 5 Indic languages, 2 datasets, and 3 ASR models. On curated FLEURS data, inflated model gaps shrink by up to 12 percent. On noisier Common Voice data, gains are smaller or mixed. That split signals true recognition weakness, not just script mismatch. Stress tests cut artificial romanization inflation by 67 percent. Sensitivity to real word errors stays near unchanged. Token collision rates stay below 0.1 percent. Pattnayak argues SN-WER should be reported beside WER and CER.

Catalog record

CREATORPriyaranjan Pattnayak
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02548

Browse the full library · About the Tower of Babel