LATINX: ALIGNING A MULTILINGUAL TTS MODEL WITH DIRECT PREFERENCE OPTIMIZATION

Luís Felipe Chary, Miguel Arjona Ramírez · 2025

Read LATINX: ALIGNING A MULTILINGUAL TTS MODEL WITH DIRECT PREFERENCE OPTIMIZATION by Luís Felipe Chary, Miguel Arjona Ramírez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05863.txt.

Introduction

Keeping a speaker's voice while translating their speech into another language is the goal of LatinX, a 2025 text-to-speech model from Luis Felipe Chary and Miguel Arjona Ramirez. The model serves cascaded speech translation, where speech is recognized, translated, then spoken again. Losing the original voice in that chain makes output feel impersonal.

LatinX is a twelve-layer decoder-only transformer trained in three stages: text-to-audio pretraining, fine-tuning for zero-shot voice cloning, and preference alignment scored by word error rate and speaker similarity. Trained on English and Romance languages with emphasis on Portuguese, the aligned model lowered word errors and raised similarity scores over its fine-tuned base. Human listeners rated its speaker match above a strong rival system, even where objective scores disagreed, and the authors discuss faster, better-balanced versions as next steps.

Catalog record

CREATORLuís Felipe Chary, Miguel Arjona Ramírez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-05863

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LATINX: ALIGNING A MULTILINGUAL TTS MODEL WITH DIRECT PREFERENCE OPTIMIZATION Luís Felipe Chary, Miguel Arjona Ramírez Abstract We present LatinX, a multilingual text-to-speech (TTS) model for cascaded speech-to-speech translation that preserves the source speaker’s identity across languages. LatinX is a 12-layer decoder-only Transformer trained in three stages: (i) pre-training for text-to-audio mapping, (ii) supervised fine-tuning for zero-shot voice cloning, and (iii) alignment with Direct Preference Optimization (DPO) using automatically labeled pairs based on Word Error Rate (WER) and speaker-similarity metrics. Trained on English and Romance languages with emphasis on Portuguese, LatinX with DPO consistently reduces WER and improves objective similarity over the fine-tuned baseline. Human evaluations further indicate stronger perceived speaker similarity than a strong baseline (XTTSv2), revealing gaps between objective and subjective measures. We provide cross-lingual analyses and discuss balanced preference signals and lower-latency architectures as future work. † † address: Escola Politécnica, Universidade de São Paulo São Paulo, Brazil {luisfchary, maramire}@usp.br 1 Introduction Despite increasing global connectivity, language barriers remain a significant challenge, particularly in regions like Brazil where fluency in English is limited [ 1 ] . While text translation tools have advanced, they fail to convey the rich paralinguistic information embedded in speech, such as emotion, tone, and vocal identity. State-of-the-art speech-to-speech (S2S) translation systems aim to bridge this gap, but preserving the original speaker’s voice across languages remains a complex task. Recent advancements in generative audio, particularly autoregressive models like VALL-E [ 2 ] that operate on discrete audio tokens, have shown remarkable capabilities in zero-shot Text-to-Speech (TTS). This paradigm, where a model clones a speaker’s voice from a short audio prompt, provides a powerful foundation for creating more natural and personalized S2S systems. This work builds upon that foundation by developing and analyzing LatinX 1 1 1 Audio samples are available at: https://seu-usuario.github.io/latinx-demo , the core TTS component of a cascaded pipeline for multilingual, voice-preserving speech-to-speech translation. The core of our system is a decoder-only Transformer model trained in a three-stage process. The key contributions of this paper are: • The development of a complete, modular pipeline for multilingual TTS, featuring custom G2P and neural codec components; • A detailed methodology for aligning the generative model using Direct Preference Optimization (DPO) [ 3 ] ; • An extensive evaluation across six languages (en, es, fr, it, pt, ro), including a granular cross-lingual analysis; • A discussion on the notable divergence between objective and subjective evaluation metrics for speaker similarity. 2 Data and Pre-processing The training corpus was compiled from three sources: Common Voice (v16.1, Dec. 2024) [ 4 ] , Multilingual Librispeech [ 5 ] , and a private collection assembled by the authors to increase the representation of Portuguese. We used data for six languages: English (en), Spanish (es), French (fr), Italian (it), Portuguese (pt), and Romanian (ro). From an initial set of over 16,500 hours, we applied a strict filtering pipeline to ensure data quality. First, transcriptions containing non-phonetic characters or artifacts were removed. Second, audio quality was assessed using TorchAudio-Squim [ 6 ] , and we retained only samples meeting the following criteria: Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) \geq 10 , Perceptual Evaluation of Speech Quality (PESQ) \geq 2.5 , and Short-Time Objective Intelligibility (STOI) \geq 0.8 . The final filtered dataset comprises approximately 9,736 hours of audio from over 84,000 unique speakers. This dataset is intentionally unbalanced, with Portuguese accounting for roughly 3,844 hours (approx. 40% of the total), to foster a strong base model for this language. The linguistic imbalance is addressed during the fine-tuning stage via sub-sampling techniques. All audio files were resampled to 16 kHz. For the Grapheme-to-Phoneme (G2P) conversion, we utilized phonetic dictionaries from the ‘ipa-dict‘ repository [ 7 ] . A summary of the data composition before and after filtering is presented in Table 1 . Table 1: Training data composition by language, showing total duration and number of unique speakers after quality filtering. Language Duration (hours) Speakers Original Filtered Original Filtered English (en) 3,490.4 2,071.8 70,258 53,490 Portuguese (pt) 7,731.5 3,844.1 2,991 2,386 French (fr) 2,143.7 1,643.8 16,173 12,237 Spanish (es) 2,174.9 1,494.2 16,381 11,371 Italian (it) 963.6 663.6 6,580 4,936 Romanian (ro) 25.1 18.3 291 268 Total 16,529.2 9,735.7 112,674 84,688 3 Model Architecture LatinX is a modular pipeline composed of three main components. 3.1 Grapheme-to-Phoneme (G2P) Model To provide a robust phonetic input representation, we use LatPhon [ 8 ] , a lightweight Transformer-based encoder-decoder model specifically trained for Romance languages and English. 3.2 Neural Audio Codec and Vocoder We employ a neural audio codec, Spectrogram Patch Codec [ 9 ] , to discretize speech. Its VQ-VAE component [ 10 ] transforms mel-spectrograms into a sequence of discrete indices from a codebook of 4096 vectors, using an encoder with a downsampling factor of 4 and a latent dimension of 512. The model is trained with a combination of a reconstruction loss ( l_{1} + LPIPS), an adversarial loss using a PatchGAN discriminator, and a vector quantization commitment loss. A HiFi-GAN vocoder [ 11 ] then converts the reconstructed mel-spectrograms back into a high-fidelity waveform. 3.3 Autoregressive Generation Model: LatinX The core of our system is a 12-layer decoder-only Transformer with 210M parameters, inspired by LLaMA 2 [ 12 ] . The architecture features an embedding dimension of 1024, a 4096-dimensional feed-forward network, and 16 attention heads. For efficiency, we incorporate Rotary Position Embeddings (RoPE) [ 13 ] and FlashAttention [ 14 ] . The model operates on a fixed context window of 8192 tokens, partitioned into three segments: 256 tokens for the input phonemes, 3968 tokens for the acoustic prompt (from a 3-second voice sample), and 3968 tokens for the autoregressive generation. It is trained to predict the subsequent audio token based on this combined context. On an NVIDIA RTX 4090 GPU with float16 precision, the model achieves an inference speed of approximately 130 tokens per second. Given that our neural codec produces roughly 630 tokens per second of audio, this corresponds to a Real-Time Factor (RTF) of approximately 4.85, making it suitable for offline synthesis. Unless otherwise stated, we use repetition-aware sampling [ 15 ] with top- p{=}1.0 , temperature T{=}0.7 , repetition window size W{=}240 and max repetition M{=}10 . 4 Training and Optimization The LatinX model is trained in three sequential stages (fig. 1 ) using the AdamW optimizer with \beta_{1}=0.9 , \beta_{2}=0.95 , and a weight decay of 0.01. Figure 1: The three-stage training pipeline for the LatinX model, progressing from general pre-training to supervised fine-tuning for voice cloning, and concluding with DPO alignment. 4.1 Stage 1: Pre-training The model is first trained for 400,000 steps on the full dataset with a standard cross-entropy loss objective. This stage teaches the model the fundamental mapping from phonemic sequences to corresponding audio codes, without any speaker conditioning. A cosine decay learning rate schedule with warmup was used. 4.2 Stage 2: Supervised Fine-tuning Next, we fine-tune the model for zero-shot voice cloning over 30,000 steps. The training data consists of triplets (\text{audio}_{\text{context}},\text{text},\text{audio}_{\text{target}}) where all audio originates from the same speaker. To construct these triplets, we enforced a minimum cosine similarity of 0.6 between speaker embeddings of any two utterances. This threshold serves a dual purpose: first, it ensures high intra-speaker acoustic consistency

Read the complete text