From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training

Tianqiao Liu, Xueyi Li, Hao Wang, Haoxuan Li, Zhichao Chen, Weiqi Luo, Zitao Liu · 2025

Read From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training by Tianqiao Liu, Xueyi Li, Hao Wang, Haoxuan Li, Zhichao Chen, Weiqi Luo, Zitao Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20072.txt.

Introduction

Speech-to-speech chat systems are the topic of this 2025 arXiv paper by Tianqiao Liu and six co-authors. These systems extend large language models so they can talk, not just type. Most current systems mix audio and text but generate both in the same step-by-step way. The authors argue that text and audio depend on different links, so one shared method fits poorly.

Their system, called Text-to-Talk, joins two styles in one Transformer model. Text is still generated step by step, while audio is built by a diffusion process that works in parallel. A special attention design keeps text causal but lets audio spans look both ways. Three training tricks reduce the gap between training and real use. On question answering, speech recognition, captioning, and speech chat tests, the system beat strong baselines. The team also plans to release its models, data, and code.

Catalog record

CREATORTianqiao Liu, Xueyi Li, Hao Wang, Haoxuan Li, Zhichao Chen, Weiqi Luo, Zitao Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20072

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training Tianqiao Liu, Xueyi Li, Hao Wang, Haoxuan Li, Zhichao Chen, Weiqi Luo, Zitao Liu Abstract Recent advances in large language models (LLMs) have attracted significant interest in extending their capabilities to multimodal scenarios, particularly for speech-to-speech conversational systems. However, existing multimodal models handling interleaved audio and text rely on autoregressive methods, overlooking that text depends on target-target relations whereas audio depends mainly on source-target relations. In this work, we propose Text-to-Talk (TtT), a unified audio-text framework that integrates autoregressive (AR) text generation with non-autoregressive (NAR) audio diffusion in a single Transformer. By leveraging the any-order autoregressive property of absorbing discrete diffusion, our approach provides a unified training objective for text and audio. To support this hybrid generation paradigm, we design a modality-aware attention mechanism that enforces causal decoding for text while allowing bidirectional modeling within audio spans, and further introduce three training strategies that reduce train-test discrepancies. During inference, TtT employs block-wise diffusion to synthesize audio in parallel while flexibly handling variable-length outputs. Extensive experiments across Audio-QA and ASR tasks demonstrate the effectiveness of our approach, with detailed ablation studies validating each proposed component. We will open-source our models, data and code to facilitate future research in this direction. 1 Introduction The recent success of LLMs has catalyzed a paradigm shift towards general-purpose Multimodal Large Language Models (MLLMs) capable of processing and generating information across diverse modalities ( Xu et al. 2025 ; Team et al. 2023 ) . Among these, speech-in/speech-out conversational systems have emerged as a pivotal component in facilitating natural human-AI interaction. Conventional systems typically decompose this problem into a cascaded pipeline of automatic speech recognition (ASR), LLM-driven response generation, and text-to-speech (TTS) synthesis. While effective to a degree, this modular design introduces significant latency accumulation and error propagation between modules, hindering naturalness and real-world applicability. In response, recent end-to-end approaches like Moshi Défossez et al. 2024 , GLM4-Voice Zeng et al. 2024 , and VITA-Audio Long et al. 2025 have sought to unify speech understanding and generation within a single model. These models are typically trained through multi-stage pipelines that involve text-to-audio tokenizer training, interleaved data construction, text-audio alignment and task-oriented supervised fine-tuning ( Huang et al. 2025 ; Li et al. 2025 ; Ding et al. 2025 ; Chu et al. 2024 ) . As shown in Fig. 1 , these methods aim to generate interleaved text and speech tokens in an autoregressive manner, which are then decoded into continuous audio waveforms by a separate neural codec or diffusion-based decoder ( Mehta et al. 2024 ; Kong et al. 2020 ) . Figure 1: (a) Distinct dependency structures for text and audio modality. (b) Due to disparate tokenization rates, the last audio span is of variable length. However, this emerging paradigm faces a fundamental challenges. As illustrated in Figure 1 , we identify a fundamental mismatch in prevailing approaches that employ a single language model to autoregressively generate both text and audio tokens. This uniform treatment applies identical autoregressive training objectives across both modalities, overlooks a critical distinction in their underlying generative processes ( Zeng et al. 2024 ; Xie & Wu 2024 ; Borsos et al. 2023 ; Dang et al. 2024 ; Rubenstein et al. 2023 ) . Text generation inherently follows a sequential causal structure characterized by strong target-target dependencies Box et al. 2015 , where each token explicitly conditions on previously generated tokens. Consequently, an incorrect token prediction can propagate and introduce subsequent errors due to the exposure bias inherent in AR models Ranzato et al. 2015 . In contrast, audio token generation is predominantly driven by source-target dependencies Ren et al. 2020 , where audio output primarily condition on the source text rather than on the preceding audio tokens. Specifically, within the current NAR span, audio tokens generation should remain faithful to the source text even when previous audio tokens are incorrectly predicted. Applying a purely AR objective to audio generation thus introduces unnecessary sequential constraints, leading to suboptimal training dynamics and magnifying error propagation. This problem can be substantially alleviated by adopting a non-autoregressive generation strategy, which aligns better with the source-dependent nature of audio modeling. Recently, discrete diffusion has emerged as a compelling alternative to AR for discrete sequence modeling ( Yu et al. 2025 ; Gong et al. 2024 ; Austin et al. 2021 ; Sahoo et al. 2024 ) . Beyond empirical gains, recent theory shows that absorbing discrete diffusion can be interpreted as modeling the conditional distributions of clean tokens and admits a tight connection to any‑order AR objectives Ou et al. 2024 . Thus, we introduce Text‑to‑Talk (TtT), a unified audio‑text MLLM that integrates AR text generation with NAR audio diffusion within a single Transformer initialized from a pretrained LLM. Text segments are trained with a standard AR cross-entropy objective, while audio segments are modeled via an NAR discrete diffusion process. During inference, the model dynamically switches between AR and NAR decoding strategies based on special control tokens. In summary, our work makes the following contributions: • We identify and formalize the fundamental asymmetry in dependency structures between text and audio modalities—text exhibits target-target dependencies requiring causal ordering, while audio is driven by source-target dependencies. Leveraging the any-order autoregressive nature of absorbing discrete diffusion, we establish a unified theoretical framework that proves our joint training objective provides an upper bound on the negative log-likelihood of the desired joint distribution. • We propose TtT, a hybrid AR-NAR MLLM that seamlessly integrates autoregressive text generation with discrete diffusion-based audio synthesis within a single Transformer initialized from a pretrained LLM. Our design preserves the reasoning and instruction-following capabilities of the base LLM while enabling efficient parallel audio generation. • We introduce three principled training strategies to address the inherent train-test discrepancies in hybrid AR-NAR learning, enabling stable training and robust content-aware variable-length generation that bridges the gap between training and inference conditions. • Extensive experiments on ASR and Audio-QA benchmarks show that our proposed TtT framework substantially outperforms strong autoregressive (AR) and non-autoregressive (NAR) baselines. Our results underscore the superiority of the hybrid AR-NAR architecture, and detailed ablations validate the effectiveness of our specialized training strategies. 2 Preliminary and notation In this section, we establish the basic notation for interleaved audio-text sequences and provide brief overviews of the two core generative paradigms employed in our framework: AR modeling and absorbing discrete diffusion. These form the theoretical foundation of our proposed method in Section 3 . Tokens, vocabulary, and interleaved layout. We consider interleaved discrete text–audio sequences of length L : x=(x^{1},\ldots,x^{L}) with a unified discrete vocabulary \mathcal{V}=\mathcal{V}_{\text{text}}\cup\mathcal{V}_{\text{audio}}\cup\mathcal{S} , where \mathcal{S} contains special tokens such as \langle\text{SOA}\rangle (start of audio), \langle\text{EOA}\rangle (end of audio), \langle\text{EOS}\rangle (end of sequence) and the absorbing mask token [\mathbf{M}] . A sequence x is structured as a series of alternating text and audio spans: x=(\mathcal{T}_{1},\mathcal{A}_{1},\ldots,\mathcal{T}_{M},\mathcal{A}_{M},\langle\text{EOS}\rangle) , where: • \mathcal{T}_{m}=(t_{m,1},\ldots,t_{m,|\mathcal{T}_{m}|})\in(\mathcal{V}_{\text{text}}\cup\{\langle\text{EOS}\rangle,\langle\text{SOA}\rangle\})^{|\mathcal{T}_{m}|} are text tokens • \mathcal{A}_{m}=(a_{m,1},\ldots,a_{m,|\mathcal{A}_{m}|})\in(\mathcal{V}_{\text{audio}}\cup\{\langle\text{EOA}\rangle\})^{|\mathcal{A}_{m}|} are quantized audio tokens Let f_{\theta}:\mathcal{V}^{L}\to\mathbb{R}^{L\times d} be a single Transformer (e.g. Qwen 2.5). We use a shared output head W\in\mathbb{R}^{d\times|\mathcal{V}|} (typically tied with input embeddings) to produce per-position logits over the entire vocabulary \mathcal{V} . 2.1 Autoregressive Modeling Autoregressive (AR) models are a fundamental class of generative models

Read the complete text