CN-NEWSTTS BENCH: A TARGET-LEVEL AUTOMATIC BENCHMARK FOR RAW-INPUT CHINESE NEWS TTS PRONUNCIATION

Shijun Luo \sthanks First author and corresponding author · 2026

Read CN-NEWSTTS BENCH: A TARGET-LEVEL AUTOMATIC BENCHMARK FOR RAW-INPUT CHINESE NEWS TTS PRONUNCIATION by Shijun Luo \sthanks First author and corresponding author (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24714.txt.

Introduction

Shijun Luo introduces CN NewsTTS Bench, an open benchmark for Chinese news text to speech. News writing is dense with tricky written forms. Percentages, unit symbols, ranges, hyphenated model names, scores, and English abbreviations all appear. A speech system can keep the written string yet speak the wrong meaning, reading a score as a range, for example. The benchmark tests whether products can pronounce such targets correctly from raw text, with no rules, rewriting, or markup hints allowed.

The first release ships a development set of 200 records and a test set of 800. It holds 992 public targets that an automatic scorer can check. Transcripts come from an ensemble of three speech recognition systems. Luo reports first results for seven product speech systems. The best reaches about 88 percent strict accuracy. Several others stay below 60 percent. The release also includes route diagnostics, subset ablations, results by category, confidence intervals, and provider settings.

Catalog record

CREATORShijun Luo \sthanks First author and corresponding author
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CN-NEWSTTS BENCH: A TARGET-LEVEL AUTOMATIC BENCHMARK FOR RAW-INPUT CHINESE NEWS TTS PRONUNCIATION Shijun Luo \sthanks First author and corresponding author Abstract Chinese news text contains dense written forms such as scores, hyphenated model names, ranges, unit symbols, percentages, English abbreviations, and mixed Chinese–Latin–digit names. These forms are frequent in real listening workflows, and a text-to-speech (TTS) system can preserve the written string while changing the spoken meaning. We introduce CN-NewsTTS Bench v0.1, an open target-level benchmark for evaluating whether Chinese news TTS products pronounce such targets correctly from raw text, without user-side rules, LLM rewriting, SSML hints, or manual edits. The release contains a 200-record development set, an 800-record public test set, 992 public auto-evaluable targets, fixed transcripts from a three-ASR ensemble, an automatic target scorer, and initial results for seven product TTS systems. We additionally report ASR-route diagnostics, ASR-subset ablations, category-level results, confidence intervals, and provider configuration metadata. The best system reaches 0.879 strict accuracy, while several systems remain below 0.60. [ BoldFont=texgyretermes-bold.otf, ItalicFont=texgyretermes-italic.otf, BoldItalicFont=texgyretermes-bolditalic.otf ] [ BoldFont=texgyreheros-bold.otf, ItalicFont=texgyreheros-italic.otf, BoldItalicFont=texgyreheros-bolditalic.otf ] [ BoldFont=texgyrecursor-bold.otf, ItalicFont=texgyrecursor-italic.otf, BoldItalicFont=texgyrecursor-bolditalic.otf ] \setCJKmainfont FandolSong-Regular.otf[ BoldFont=FandolSong-Bold.otf, ItalicFont=FandolKai-Regular.otf ] \setCJKmonofont FandolSong-Regular.otf † † address: NetEase Cloud Music, Hangzhou, China 1 Introduction Modern Chinese news TTS systems are used in settings where listeners expect symbols and compact written forms to be read according to news conventions. A short article may contain a sports score such as \xeCJKsetup CJKecglue= 96-91 , an aircraft model such as \xeCJKsetup CJKecglue= 苏-27 , a year range such as \xeCJKsetup CJKecglue= 2028-2030年 , a torque unit such as \xeCJKsetup CJKecglue= 620N \cdot m , or a generation label such as \xeCJKsetup CJKecglue= 80后 . These strings are usually unambiguous to a human editor, yet the same surface marks can represent ranges, subtraction, model designators, unit names, or ordinary numbers. Such failures are not merely naturalness problems. Reading a score as a range changes the relation between the numbers; reading \xeCJKsetup CJKecglue= 苏-27 as a negative number changes a military model into a nonsensical phrase; reading \xeCJKsetup CJKecglue= 80后 as \xeCJKsetup CJKecglue= 八十后 loses the conventional generation-label reading. Subjective tests such as MOS remain important for speech quality [ 5 ] , and objective predictors can estimate global naturalness or voice-conversion quality [ 6 ] . They are, however, too coarse to track specific high-impact reading decisions. Generic ASR round-trip comparison is also insufficient because the evaluation must decide whether a particular target was read as an intended news form. The problem is closely related to text normalization and non-standard-word handling in TTS [ 10 , 8 , 2 , 9 ] , Mandarin text normalization [ 12 , 1 ] , and Chinese G2P/polyphone benchmarks [ 7 , 11 ] . CN-NewsTTS Bench differs by measuring the end-to-end raw-input behavior exposed by deployed TTS APIs rather than an isolated frontend component. The main contributions are: • an open, deterministic dev/public split for Chinese news pronunciation targets; • a Raw Input Product Track that excludes user-side normalization or hints; • a target-level schema with positive and negative readings; • a three-ASR automatic scoring protocol with route diagnostics and ablations; and • a reproducible v0.1 leaderboard for seven product TTS systems. 2 Benchmark Design 2.1 Raw Input Product Track The main track measures product-facing behavior under a fixed raw-input condition. Each system receives the same original Chinese news-style text. Provider-internal text normalization and frontend behavior are allowed, but external rule frontends, LLM rewrites, SSML pronunciation hints, and manual benchmark-text edits are not allowed. A fixed voice is chosen before evaluation and kept constant for all samples. This is therefore a raw-input product evaluation, not a claim that the benchmark isolates model weights or removes all provider-side frontend logic. 2.2 Data Construction and Targets The v0.1 data are deterministic synthetic news-style sentences generated from category-specific templates and lexicons, not reused copyrighted news articles. The released generator script uses seed 20260620, 11 template families, and 86 lexicon entries covering teams, events, projects, cities, companies, military models, vehicle models, abbreviations, mixed brand strings, unit symbols, generation labels, and polyphone audit items. The templates mirror production risk motifs such as sports recaps, defense/vehicle models, technical units, mixed brands, and time spans; they are diagnostic rather than a claim about full news frequency. The dev and public splits are constructed from fixed per-category quotas, then shuffled. Each record contains targets with offsets, categories, positive/negative readings, and an auto-evaluable flag. Optional same-character polyphones are included for audit but excluded from the main automatic score because ordinary ASR text often hides tonal distinctions. Table 1. CN-NewsTTS Bench v0.1 data scale. Split Rec. Targets Auto Opt. dev 200 252 248 4 public 800 1008 992 16 total 1000 1260 1240 20 Table 2: Representative target annotations. Positive and negative patterns are simplified examples; the released JSONL contains the full pattern lists. Span Type Positive Negative \xeCJKsetup CJKecglue= 96-91 score \xeCJKsetup CJKecglue= 九十六比九十一 \xeCJKsetup CJKecglue= 九十六到九十一 \xeCJKsetup CJKecglue= 苏-27 model \xeCJKsetup CJKecglue= 苏二七 ; \xeCJKsetup CJKecglue= 苏二十七 \xeCJKsetup CJKecglue= 苏负二十七 \xeCJKsetup CJKecglue= 80后 generation \xeCJKsetup CJKecglue= 八零后 \xeCJKsetup CJKecglue= 八十后 \xeCJKsetup CJKecglue= 620N \cdot m unit \xeCJKsetup CJKecglue= 六百二十牛米 \xeCJKsetup CJKecglue= 六百二十恩点米 The public split contains 224 sports-score targets, 192 military-model targets, 104 abbreviations, 96 unit-symbol targets, 80 hyphenated ranges, 80 year ranges, 64 mixed brand names, 64 vehicle models, 56 percentages, and 32 generation labels. The public test set is a transparent diagnostic split rather than a long-term anti-overfitting competition set; a hidden split can be added if leaderboard gaming becomes a practical concern. 3 Automatic Evaluation 3.1 Three-ASR Protocol The v0.1 public leaderboard uses three heterogeneous ASR routes: MiMo API ASR, SenseVoiceSmall, and Paraformer-zh. SenseVoiceSmall and Paraformer-zh are open-source local recognizers from the FunAudioLLM/FunASR ecosystem [ 4 , 3 ] ; MiMo API ASR provides an API-based route. The public release fixes all three ASR transcript files, so the leaderboard can be reproduced without rerunning commercial TTS APIs or ASR services. For seven TTS systems and 800 public records, the canonical transcript set contains 16,800 successful ASR rows and no missing public samples. 3.2 Target-Level Voting and Metrics For each ASR transcript and target, the scorer applies Unicode, case, punctuation, and whitespace normalization. It then matches negative-reading patterns before positive-reading patterns. A single ASR route returns wrong if a negative reading is found, correct if a positive reading is found, and unknown otherwise. Majority voting gives final correct or wrong if at least two routes agree; all remaining combinations become unknown . The primary metric is strict auto accuracy, \mathrm{StrictAcc}=\frac{\#\mathrm{correct}}{\#\mathrm{auto\mbox{-}evaluable\ targets}}. (1) We also report coverage and resolved accuracy: \displaystyle\mathrm{Coverage} \displaystyle=\frac{\#\mathrm{correct}+\#\mathrm{wrong}}{\#\mathrm{auto\mbox{-}evaluable\ targets}}, (2) \displaystyle\mathrm{ResolvedAcc} \displaystyle=\frac{\#\mathrm{correct}}{\#\mathrm{correct}+\#\mathrm{wrong}}. (3) Keeping unknowns in the strict denominator prevents a system from appearing accurate only on the subset that the ASR ensemble can resolve. 3.3 ASR Protocol Diagnostics Table 3 pools all 6,944 public target instances (7 systems \times 992 targets). The three individual routes have similar strict accuracy but different coverage behavior. MiMo API ASR is conservative: it has high resolved accuracy when it resolves a target, but many unknowns. Paraformer-zh and SenseVoiceSmall resolve more targets. Pairwise tri-label agreement is 0.959 between Paraformer-zh and SenseVoiceSmall, but 0.595 and 0.604 between MiMo API ASR and the two local routes, showing that the ensemble is heterogeneous. Overall, 41.7% of

Read the complete text