Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers

Runyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, Xiaodong Zeng · 2026

Read Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers by Runyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, Xiaodong Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10770.txt.

Introduction

Runyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, and Xiaodong Zeng present GPA, a single speech model for many tasks. Speech systems have long used separate models for text-to-speech, speech recognition, and voice conversion. Split pipelines limit scale, efficiency, and transfer between tasks. GPA instead uses autoregressive transformers for all three jobs at once.

The name plays on the school grade average: the goal is solid results across all audio subjects. One shared model can pool what each task teaches the others. Recognition may sharpen synthesis, and conversion may sharpen both. The work speaks to teams that want one speech engine for products instead of three narrow ones.

Catalog record

CREATORRunyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, Xiaodong Zeng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10770

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers Runyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, Xiaodong Zeng Abstract Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-Purpose Audio (GPA) * * * Like an academic GPA that reflects capability across subjects, GPA aims for decent results across all audio tasks. , a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture. GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly perform TTS, ASR, and VC without architectural modifications. This unified design combines a fully autoregressive formulation over discrete speech tokens, joint multi-task training across speech domains, and a scalable inference pipeline that achieves high concurrency and throughput. The resulting model family supports efficient multi-scale deployment, including a lightweight 0.3B-parameter variant optimized for edge and resource-constrained environments. Together, these design choices demonstrate that a unified autoregressive architecture can achieve competitive performance across diverse speech tasks while remaining viable for low-latency, practical deployment. Keywords Text-to-Speech \cdot Automatic Speech Recognition \cdot Voice Conversion \cdot Foundation Model 1 Introduction The rapid advancement of generative artificial intelligence has fundamentally reshaped the landscape of speech processing. Driven by the success of Large Language Models (LLMs) in natural language processing, the speech community has moved from specialized, task-specific pipelines to large-scale, data-driven foundation models. Significant breakthroughs have been achieved in core tasks such as Text-to-Speech (TTS), Automatic Speech Recognition (ASR), and Voice Conversion (VC), where neural approaches now demonstrate human-level performance in distinct scenarios [ 49 , 38 , 31 ] . Despite these individual successes, the current ecosystem remains architecturally fragmented. While TTS, ASR, and VC share intrinsic linguistic and acoustic foundations, they are typically treated as isolated problems, each relying on distinct modeling paradigms and architectural assumptions: • Text-to-Speech: Modern zero-shot TTS systems have evolved into diverse categories, ranging from discrete token-based language models that autoregressively generate acoustic tokens [ 45 , 22 , 42 , 48 ] , to diffusion-based probabilistic mappings that implicitly learn alignments between text and speech [ 25 , 3 ] , as well as coarse-to-fine hybrid architectures that combine semantic modeling with non-autoregressive acoustic rendering [ 19 , 13 , 15 ] . While hybrid models currently offer a favorable balance between synthesis quality and controllability, their multi-stage design introduces significant system complexity, hindering unified modeling and efficient deployment. • Automatic Speech Recognition: ASR has rapidly transitioned toward large-scale, pre-trained encoders and encoder–decoder models, such as Whisper [ 38 ] , Qwen-Audio [ 11 , 10 ] , SenseVoice [ 1 ] , and Seed-ASR [ 6 ] , demonstrating strong multilingual and multitask capabilities. However, these models are primarily optimized for discriminative recognition objectives and are architecturally specialized for ASR, making them difficult to repurpose for generative or conversion-oriented speech tasks without substantial architectural modification or auxiliary components [ 51 , 40 , 27 , 47 , 52 ] . • Voice Conversion: VC systems generally rely on disentanglement-based frameworks that factorize speech into linguistic content, timbre, and speaking style to enable controllable attribute manipulation [ 36 , 34 ] . Existing approaches include parallel-data-based conversion [ 60 , 20 ] , disentangled representation learning [ 64 , 14 ] , and large-scale in-context learning [ 46 , 26 , 54 ] . Despite these advances, VC methods often suffer from limited scalability and imperfect decoupling of attributes, frequently requiring parallel corpora, explicit style annotations, or additional fine-tuning stages to achieve robust zero-shot generalization [ 21 , 61 , 35 , 59 ] . This fragmentation results in duplicated modeling efforts, limited cross-task knowledge transfer, and significant deployment complexity. To date, there is a lack of a fully unified generative framework that can natively support speech understanding, generation, and conversion within a single autoregressive modeling and inference pipeline , without resorting to task-specific heads or routing mechanisms. In this work, we present General-Purpose Audio (GPA), a unified speech foundation model that integrates TTS, ASR, and VC into a single autoregressive LLM framework. Unlike multi-stage or hybrid systems, GPA formulates all speech tasks as sequence modeling problems over a shared discrete audio token space. By conditioning on task-specific instructions, a single model can seamlessly switch between recognizing speech, synthesizing audio, and converting voices without architectural modification. The main contributions of this work are summarized as follows: • Unified Autoregressive Audio Framework: We propose GPA, a unified autoregressive LLM framework for TTS, ASR, and VC. GPA employs a dual-tokenizer scheme, leveraging both GLM tokenizer [ 55 ] and BiCodec tokenizer [ 48 ] for speech discretization. By placing all tasks within a shared discrete audio token space, GPA enables instruction-driven task switching without any architectural modifications. • Synergistic Multi-Task Learning: We show that joint training across multiple tasks—using the Emilia [ 18 ] dataset and supplementary proprietary data—enhances overall model performance. This unified approach allows ASR, TTS, and VC to benefit from shared representations in a discrete latent space, outperforming training each task in islation. • High-Throughput and Streaming Efficiency: We demonstrate that purely autoregressive architecture of GPA enables straightforward deployment, superior concurrency, and higher overall throughput compared to hybrid alternatives. This design choice makes GPA well-suited for streaming applications and high-demand inference scenarios. • Edge-Optimized Scalability: We introduce a family of GPA models spanning multiple scales, including a highly compact 0.3B-parameter variant explicitly designed for deployment in resource-constrained and edge environments. This enables multi-task on-device inference within a single model instance while maintaining a minimal memory footprint. 2 General Purpose Audio Speech understanding, generation, and editing tasks such as ASR, TTS, and VC are commonly treated as distinct problems and addressed using task-specific pipelines. While effective, these designs fragment modeling choices and often rely on heterogeneous components. Although recent advances in LLMs have driven progress across speech tasks, hybrid approaches [ 15 ] that depend on downstream synthesis modules [ 9 ] [ 32 ] remain constrained by latency and system efficiency. In contrast, the BiCodec system [ 48 ] provides empirical evidence that a purely autoregressive architecture can achieve strong performance in TTS task. This success suggests that when speech and text are represented in a shared discrete token space, diverse audio tasks can be uniformly formulated as sequence-to-sequence generation problems, differing only in their input–output specifications. Based on this insight, we arrive at a unifying principle: all speech tasks admit a single autoregressive formulation in a fully discrete token space. Figure 1 presents the architecture of our proposed GPA framework. Building on Qwen3 [ 53 ] LLM backbone, GPA adopts the BiCodec framework and integrates an extra GLM [ 55 ] tokenizer to enhance semantic abstraction. By unifying textual, semantic, and acoustic information into a shared discrete token space, ASR, TTS, and VC can naturally emerge as instances of a single sequence generation task. This results in a general-purpose audio framework based solely on autoregressive generation, with task behavior controlled entirely by instructions— no task-specific architectures or decoding pipelines required. Figure 1: Architecture of the proposed

Read the complete text