RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matching

Jinhyeok, Hyeongju, Yechan, Joon, Frederik, Juheon · 2026

Read RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matching by Jinhyeok, Hyeongju, Yechan, Joon, Frederik, Juheon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22083.txt.

Introduction

Jinhyeok Yang and coauthors present RobustSpeechFlow, a training method that makes flow-matching text-to-speech more robust to alignment errors. Flow-matching speech models achieve zero-shot speaker similarity and naturalness, yet they suffer fidelity problems, notably skip and repeat errors from imperfect alignment. RobustSpeechFlow extends contrastive flow matching using length-preserving repeat and skip augmentations, needing no external aligners or preference data. On Seed-TTS-eval it cuts word error rate from 1.44 to 1.38 with only 0.06 billion parameters, and on ZERO500 it improves intelligibility across speaker and prosody conditions.

Catalog record

CREATORJinhyeok, Hyeongju, Yechan, Joon, Frederik, Juheon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24676
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22083

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matching Jinhyeok, Hyeongju, Yechan, Joon, Frederik, Juheon Abstract While flow-matching text-to-speech (TTS) achieves strong zero-shot speaker similarity and naturalness, it remains susceptible to content fidelity issues, particularly skip and repeat errors from imperfect alignment. We propose RobustSpeechFlow, a training strategy that improves alignment robustness by extending contrastive flow matching with length-preserving repeat and skip latent augmentations. Requiring no external aligners or preference data, our method directly penalizes realistic failure modes and readily integrates into existing pipelines. On Seed-TTS-eval, it reduces the word error rate (WER) from 1.44 to 1.38 using only 0.06B parameters. On our ZERO500 benchmark, it delivers consistent intelligibility improvements across diverse speaker and prosody conditions; at NFE=24, it reduces English character error rate (CER) from 0.48% to 0.35% and Korean CER from 0.81% to 0.57%. Audio samples: https://robustspeechflow.github.io/ . Yang Kim Yu Byun Bous Lee keywords text-to-speech, zero-shot TTS, flow matching, alignment robustness, contrastive learning † † address: 1 Supertone Inc, South Korea 2 Independent Researcher, South Korea † † email: [email protected], [email protected] 1 Introduction In text-to-speech (TTS), the primary requirement is content fidelity : the system must render the intended text accurately. In production, failures such as word or phrase repetition and skipping are not minor artifacts; they materially reduce reliability and can create safety and compliance risks. As modern generative TTS systems continue to improve naturalness and expressivity, robust text–speech alignment remains a central problem. Recent progress in non-autoregressive TTS has increasingly been driven by flow-matching-based latent generation. DiTTo-TTS shows that large-scale diffusion transformers can achieve strong zero-shot quality without relying on domain-specific factors such as phonemes or phoneme durations, highlighting the value of scalable latent modeling and cross-attention-based alignment [ 13 ] . F5-TTS further demonstrates that flow matching with a DiT backbone can simplify the pipeline by removing explicit duration modeling and alignment modules while retaining high quality and robustness [ 4 ] . At the efficiency end of the spectrum, SupertonicTTS shows that compact flow-matching TTS can be practical for deployment, but also makes clear that content stability becomes increasingly difficult under tight capacity and latency constraints [ 12 ] . In parallel, the integration of autoregressive language modeling with diffusion processes [ 21 ] has catalyzed a new wave of highly expressive speech generation. Building upon this paradigm, recent systems such as DiTAR [ 9 ] , VoxCPM [ 28 ] , and VibeVoice [ 16 ] have substantially improved zero-shot quality and contextual naturalness. While these advances broaden the scope of generative speech modeling, they do not reliably resolve the issue of content fidelity. In practice, both pure flow-matching architectures and hybrid LM-diffusion systems continue to exhibit alignment errors. Crucially, these skip and repeat failures tend to become more severe as model capacity is constrained or the number of function evaluations (NFE) is reduced during inference [ 4 , 12 , 9 ] . Furthermore, as speech generative modeling is scaled with increasingly large and diverse ASR-curated corpora such as GigaSpeech and Emilia [ 2 , 7 ] , ensuring robust quality control and alignment becomes even more critical. Several directions have been explored to mitigate this misalignment. At the architectural level, explicit conditioning and guidance can improve low-step stability; for instance, DualSpeech uses a phoneme-level latent diffusion formulation with multiple classifier-free guidance signals to stabilize generation at low NFE [ 24 ] . Similarly, stronger architectures and pretrained representations can improve overall robustness, but they do not eliminate skip and repeat failures altogether [ 13 ] . Beyond architectural modifications, preference-based approaches construct dedicated intelligibility datasets and apply post-training objectives such as DPO to penalize failure cases [ 26 , 18 ] . Other approaches introduce auxiliary supervision from recognition or metric-optimization models, for example ASR- or CTC-derived objectives that directly penalize content errors [ 14 ] . While effective, these strategies either require costly data curation or introduce additional models and training complexity, which is undesirable for lightweight deployment settings. We propose RobustSpeechFlow , a simple training strategy for contrastive flow matching in TTS that improves alignment robustness without additional models or datasets. Our key idea is to convert common TTS failure modes into hard negatives in latent space: we synthesize skip- and repeat-like target latents with length-preserving overwrites and train the model contrastively so that the correct latent trajectory is preferred over these corrupted alternatives. This directly aligns the contrastive signal with the failure patterns that matter in practice while remaining compatible with standard flow-matching training. The main contributions are as follows. • We introduce RobustSpeechFlow, a speech-specific contrastive flow-matching training strategy that targets skip/repeat failures via simple latent-space augmentations. • The proposed method requires no preference dataset, no external aligner or ASR model, and readily integrates into FM-TTS pipelines with minimal overhead. • We demonstrate consistent improvements on both a public zero-shot benchmark and a new multilingual benchmark with higher speaker, prosody, and text diversity, especially under low-NFE inference. 2 Background 2.1 Alignment Robustness Alignment errors have been a persistent challenge since early deep-learning-based TTS. Tacotron 2 improves attention stability with location-sensitive attention [ 19 ] . Deep Convolutional TTS (DCTTS) introduces a guided-attention loss and a forced-incremental attention heuristic at synthesis time to mitigate skipped or repeated characters [ 22 ] . Despite these efforts, skip/repeat failures remain common, especially for long or out-of-distribution text. Recent work has explored post-training and auxiliary-guidance approaches. Preference-based methods construct targeted failure cases and apply direct preference optimization (DPO) training [ 26 , 18 ] . Other approaches introduce auxiliary objectives, such as CTC-based ASR supervision, to directly penalize content errors [ 14 , 6 ] . Alternatively, leveraging strong pretrained text and speech representations (e.g., ByT5 and SpeechT5) has been shown to inherently improve alignment stability, as demonstrated in DiTTo-TTS [ 13 ] . While positional encodings like LARoPE [ 11 ] provide a practical lever to improve alignment under duration variation, RobustSpeechFlow operates orthogonally: we reshape training using failure-mode contrastive negatives without introducing new models or external supervision. 2.2 Contrastive Learning and Hard Negatives Contrastive learning has been widely used to improve discriminative representations by bringing positives closer and pushing negatives apart [ 3 , 8 ] . In practice, the usefulness of the contrastive signal depends strongly on negative construction: hard negatives that remain close to the anchor but violate the conditioning semantics often provide a stronger gradient than easy, unrelated negatives. This idea has recently been extended to generative modeling. Contrastive Flow Matching (CFM) [ 20 ] , originally introduced for image generation, augments the standard conditional flow-matching objective with contrastive regularization terms computed from mismatched conditions. Concretely, it keeps the predicted vector field aligned with the positive transport direction while separating it from negative transport directions induced by incorrect conditions. This improves condition selectivity during training without changing the FM inference procedure. RobustSpeechFlow adapts this framework to speech generation and fundamentally strengthens the negative formulation for TTS. Rather than relying on random condition mismatches, we construct same-utterance latent corruptions that remain acoustically plausible while explicitly breaking local text–speech alignment. 3 RobustSpeechFlow 3.1 Preliminaries: Conditional Flow Matching Let x\in\mathbb{R}^{C\times T} be a continuous latent speech sequence produced by the Supertonic speech autoencoder [ 12 ] ,

Read the complete text