PSP : An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech

Venkata Pushpak Teja Menta · 2026

Read PSP : An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech by Venkata Pushpak Teja Menta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25476.txt.

Introduction

Menta presents PSP, an accent benchmark for Indic text-to-speech. Standard TTS evaluation gauges intelligibility and naturalness, but not accent. PSP splits accent into six dimensions: retroflex collapse, aspiration, vowel-length, Tamil-zha fidelity, Frechet Audio Distance, and prosodic divergence. Version one benchmarks five systems on Hindi, Telugu, and Tamil sets, plus an R5-to-R6 Telugu case study. Findings: retroflex collapse grows with difficulty, Hindi below Telugu below Tamil, near 1, 40, and 68 percent; PSP ordering diverges from word-error ordering; no system leads across all six dimensions.

Catalog record

CREATORVenkata Pushpak Teja Menta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39989
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25476

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PSP : An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech Venkata Pushpak Teja Menta Abstract Standard text-to-speech (TTS) evaluation measures intelligibility (WER, CER) and overall naturalness (MOS, UTMOS) but does not quantify accent . A synthesiser may score well on all four yet sound non-native on features that are phonemic in the target language. For Indic languages, these features include retroflex articulation, aspiration, vowel length, and the Tamil retroflex approximant /ṟ/ (Tamil letter ழ). We present PSP , the Phoneme Substitution Profile, an interpretable, per-phonological-dimension accent benchmark for Indic TTS. PSP decomposes accent into six complementary dimensions (retroflex collapse rate RR, aspiration fidelity AF, vowel-length fidelity LF, Tamil-zha fidelity ZF, Fréchet Audio Distance FAD in a phonetic embedding space, and prosodic signature divergence PSD), with the first four measured via forced alignment plus native-speaker-centroid acoustic probes over Wav2Vec2-XLS-R [ 2 ] layer-9 embeddings, and the latter two computed as corpus-level distributional distances. In this v1 preprint we benchmark four commercial and open-source systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) across Hindi, Telugu, and Tamil pilot sets, with a fifth system (our in-progress Praxy Voice, R6 LoRA on Te/Ta plus vanilla Chatterbox on Hi) additionally included on all three languages, including an R5 → \to R6 training-scale case study on Telugu. We report three principal findings: (i) retroflex collapse grows monotonically with phonological difficulty Hindi < < Telugu < < Tamil ( ∼ \sim 1%, ∼ \sim 40%, ∼ \sim 68%); (ii) PSP ordering diverges from WER ordering, with commercial WER-leaders not uniformly leading on retroflex or prosodic fidelity; (iii) no single system is Pareto-optimal across all six dimensions. We release native reference centroids (500 clips per language), 1000-clip utterance-level embeddings for FAD, 500-clip prosodic feature matrices for PSD, 300-utterance held-out golden sets per language, scoring code under MIT, and centroids under CC-BY. Compared to PSR [ 12 ] — a contemporary rule-based phonological benchmark for American–British English — PSP is acoustic-probe-based, Indic-specific, and per-dimension decomposed; the two are complementary rather than competing. Formal MOS-correlation calibration is deferred to v2; this v1 reports five internal-consistency signals supporting metric validity, including a native-audio sanity check that establishes a language-specific noise floor for per-phoneme probes. I Introduction Modern TTS systems for Indic languages achieve strong intelligibility: recent commercial systems report Word Error Rates (WER) below 5% for Hindi and Tamil on standard test sets, and open-source systems are closing the gap. Yet subjective listening consistently reveals a residual accent mismatch: the synthesiser pronounces every word correctly but not as a native speaker would. This paper makes the case that accent, for Indic languages, is measurable and decomposable . Native Indic phonology has systematic features — retroflex consonants contrasting with dentals, aspirated versus unaspirated stops, phonemic vowel length, the Tamil retroflex approximant — that non-native speakers routinely collapse. We propose treating accent as a vector of such per-feature substitution rates and measuring each rate via acoustic probes against native-speaker prototypes. Contributions. 1. We formalise six per-dimension accent measures for Indic TTS: retroflex collapse (RR), aspiration fidelity (AF), length fidelity (LF), Tamil-zha fidelity (ZF), Fréchet Audio Distance (FAD), and prosodic signature divergence (PSD) (§ III ). 2. We implement all six as open-source, GPU-accelerated tools, with bootstrap-resampled 95% confidence intervals computed per-system and reported in the release artefacts (§ IV ); we abstain from ranking-significance claims in v1 given small pilot n , deferring to v2 with 300-utterance scale. 3. We release native-speaker references: phoneme-centroid dictionaries (500 clips per language), 1000-clip utterance-level XLS-R embeddings, and 500-clip prosodic feature matrices, for Telugu, Hindi, and Tamil (§ IV ). 4. We benchmark four open-source and commercial systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) across Hindi, Telugu, and Tamil pilot sets, with a fifth system (our in-progress Praxy Voice) additionally included on all three languages via a language-specific routing scheme — R6 LoRA + BUPS on Te / Ta, vanilla Chatterbox on Hi — both branches sharing a BYOR voice-prompt recovery recipe; pilot sets contain 10 utterances per language (§ VI ). 5. We show five independent internal-consistency signals supporting PSP’s validity as an accent metric, deferring formal MOS calibration to v2 (§ V ). A companion paper [ 13 ] applies this benchmark as a diagnostic loop during system development; we cite that case study where its findings reinforce metric validity (§ V , § VI ). II Related work TTS quality metrics. WER and CER via ASR-based transcription remain the dominant intelligibility metrics. UTMOS [ 16 ] and MOS prediction networks (see the VoiceMOS Challenge [ 5 ] for an overview of the neural-MOS space) estimate perceived overall quality. Neither targets accent specifically. Fréchet-style distribution metrics. FAD [ 7 ] and its speech variants [ 8 ] compare embedding distributions of synthesised and reference audio. They provide a single scalar quality number but are not interpretable per phonological feature. The nPVI [ 4 ] captures syllable-timed vs stress-timed rhythmic class and serves as one of our 5-D PSD feature dimensions. Phoneme-level evaluation. PSR [ 12 ] recently introduced the Phoneme Shift Rate for quantifying how speaker embeddings preserve versus overwrite accent-dependent phoneme mappings between American and British English. PSP is a conceptual sibling, but targets a different setting: PSR is rule-based (American–British phonological rules), English-specific, and produces a single scalar; PSP is acoustic-probe-based, Indic-first, and decomposes into named per-phonological-dimension rates. We view the two as complementary: PSR for English accent research, PSP for Indic. [ 11 ] uses related phonological-rule machinery for accent generation , not evaluation. Accent similarity. [ 18 ] proposes PPG-distance plus vowel-formant distance for pairwise accent similarity; these provide a single scalar per pair and do not separate phonological dimensions. Indic speech benchmarks. Rasmalai [ 17 ] and IndicVoices-R [ 1 ] release large Indic speech corpora with accent/intonation descriptors; their evaluation pipeline uses MUSHRA listening tests, not automatic metrics. FLEURS [ 3 ] provides cross-lingual speech evaluation. The IndicWav2Vec lineage [ 6 ] supplies the language-specific CTC aligners we depend on for forced alignment. PSP complements these resources as an automatic, per-dimension accent metric for Indic TTS, intended to sit alongside MUSHRA-based listening pipelines rather than replace them. III The phoneme substitution profile Formal definition. For a TTS system S and a language \ell , let \mathcal{D}=\{D_{1},\dots,D_{k}\} be a set of phonological dimensions , each D_{i} parameterised by (i) a set of native phonemes P_{i}^{\text{nat}} , (ii) a substitute phoneme set P_{i}^{\text{sub}} that non-native speakers produce instead, and (iii) an acoustic embedding \varphi:\text{audio}\to\mathbb{R}^{d} . The fidelity of S on dimension D_{i} is \displaystyle\text{PSP}_{i}(S)=\mathbb{E}_{x\sim S,\,p\in x\cap P_{i}^{\text{nat}}}\frac{\mathrm{sim}(\varphi(\tilde{x}_{p}),\mu_{i}^{\text{nat}})}{\mathrm{sim}(\varphi(\tilde{x}_{p}),\mu_{i}^{\text{nat}})+\mathrm{sim}(\varphi(\tilde{x}_{p}),\mu_{i}^{\text{sub}})} (1) where \tilde{x}_{p} is the time span of phoneme p in generated utterance x (via forced alignment), \mu_{i}^{\text{nat}} and \mu_{i}^{\text{sub}} are native-speaker and substitute-speaker centroids for D_{i} , and \mathrm{sim} is rectified cosine similarity. Indic dimensions. For \ell\in\{ Telugu, Hindi, Tamil \} we instantiate four per-phoneme dimensions and two corpus-level dimensions. The four per-phoneme probes: • D_{\text{RR}} (Retroflex): P^{\text{nat}}=\{ ṭ, ḍ, ṇ, ṣ, ḷ \} ; P^{\text{sub}}=\{t,d,n,s,l\} . • D_{\text{AF}} (Aspiration; Hindi primary, Telugu sparse, Tamil N/A): P^{\text{nat}}=\{kh,gh,ph,bh,\dots\} ; P^{\text{sub}}=\{k,g,p,b,\dots\} . • D_{\text{LF}} (Length): P^{\text{nat}}=\{\bar{a},\bar{i},\bar{u}\} ; P^{\text{sub}}=\{a,i,u\} ; fidelity measured as a ratio comparison against a native-prior long/short duration ratio. • D_{\text{ZF}} (Tamil zha, Tamil only): P^{\text{nat}}=\{ ḻ \} ; P^{\text{sub}}=\{l\}

Read the complete text