DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment

Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao · 2025

Read DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment by Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02768.txt.

Introduction

Ke-Han Lu and co-authors present DeSTA2.5-Audio, a large model that hears and follows instructions, in a paper from 2025. Audio language models bolt hearing onto text models. Training them on sound tasks often erases the text skills they started with, a failure called catastrophic forgetting.

The team rethought how training data gets built. Their model generates its own alignment data across sound and text. That self-made data ties hearing to language without swamping old knowledge. The result aims at a general-purpose listener: strong perception plus intact instruction following. Balance between old and new skills is the paper's central problem. Such models could serve as voice assistants that truly hear their users.

Catalog record

CREATORKe-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao-Ying Huang, Huang-Cheng Chou, Kai-Wei Chang, Cheng-Han Chiang, Boris Ginsburg, Yu-Chiang Frank Wang, Hung-yi Lee Abstract We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following, without requiring task-specific audio instruction-tuning. Recent LALMs typically augment Large Language Models (LLMs) with auditory capabilities by training on large-scale, manually curated or LLM-synthesized audio-instruction datasets. However, these approaches have often suffered from the catastrophic forgetting of the LLM’s original language abilities. To address this, we revisit the data construction pipeline and propose DeSTA, a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets. This approach preserves the LLM’s native language proficiency while establishing effective audio-text alignment, thereby enabling zero-shot generalization without task-specific tuning. Using DeSTA, we construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms widely adopted data construction and training strategies in both auditory perception and instruction-following capabilities. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs. 1 1 1 https://github.com/kehanlu/DeSTA2.5-Audio I Introduction The development of general-purpose artificial intelligence has become a central focus in contemporary AI research, driven by the remarkable performance of large language models (LLMs) across various natural language understanding and generation tasks [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . Building on these advancements, a promising direction is to equip LLMs with multi-modal understanding capabilities, leading to the emergence of Large Audio Language Models (LALMs) [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ] and Large Vision Language Models (LVLMs) [ 23 , 24 , 25 , 26 , 27 ] . This paper focuses on building a general-purpose LALM, illustrated in Figure 1 . To develop a general-purpose LALM, two core capabilities are essential: auditory perception and instruction-following . Auditory perception refers to the comprehensive processing of auditory information, including speech, non-verbal cues, background sounds, and music. In addition, instruction-following entails interpreting diverse user commands, integrating them with internal knowledge, and producing contextually appropriate responses. To achieve this, most LALM architectures adopt a modular approach, combining a pre-trained audio model with a text-based LLM as the knowledge backbone [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . These components are integrated through a cross-modal alignment process, which typically involves joint fine-tuning on a corpus of paired audio and text instructions to bridge the representational gap between the audio and text modalities. Accordingly, the research problem centers on training the fusion model with limited audio-related data, while leveraging the LLM’s generalization capabilities to handle unseen tasks. Fig. 1: A general-purpose Large Audio Language Model (LALM) understands multifaceted audio information and accepts arbitrary text prompts to generate corresponding responses, enabling seamless multimodal interaction. However, acquiring high-quality data for cross-modal alignment remains a significant challenge. A common strategy involves transforming existing audio datasets by converting them into audio instruction-tuning datasets, consisting of (1) audio input, (2) textual instructions, and (3) textual responses. These training triplets can be generated either by text-based LLMs or through manual annotation. To improve the generalization capabilities of LALMs, several techniques have been adopted, including fine-tuning lightweight LoRA adapters [ 11 , 10 , 12 , 14 , 13 ] , adopting multi-stage curriculum learning [ 12 , 16 , 17 , 13 , 14 , 15 , 20 ] , and applying test-time pruning [ 12 ] . While current LALMs perform well across a range of audio-related benchmarks [ 28 , 29 , 30 , 31 , 32 , 33 , 34 ] , many studies highlight the issue of catastrophic forgetting [ 8 , 9 , 12 , 11 , 10 , 14 , 35 , 36 ] , where the models become overly specialized to a limited set of audio-related tasks at the cost of generalization to unseen tasks. Moreover, our previous work, Speech-IFEval [ 36 ] , also reveals that most LALMs experience significant degradation in text-based knowledge, underscoring the persistent limitations of current cross-modal alignment techniques. To address the issue of catastrophic forgetting in cross-modal alignment, we revisit the data construction process with a key observation: annotations from different sources, such as various LLMs or human annotators, exhibit distinct stylistic and behavioral patterns. While this may seem intuitive, prior studies often rely on responses generated by powerful LLMs [ 11 , 10 , 16 , 17 , 13 ] , such as GPT [ 1 ] , or human-designed pairs [ 12 , 14 ] to ensure high-quality supervision. However, we hypothesize that incorporating annotations from heterogeneous sources inevitably introduces biases into the dataset. As a result, the fusion model must not only learn auditory grounding but also implicitly adapt to the new response patterns introduced by the annotators. We believe this extra adaptation burden can gradually distort the LLM’s native output distribution or instruction-following behavior, ultimately degrading its text-based knowledge. Fig. 2: (Left) Dataset Construction: An audio description x^{\text{text}} and a randomly sampled prompt \mathbf{p} are fed into the backbone LLM to generate training targets \mathbf{y} . (Right) Model Training: The fusion model is trained using the self-generated targets \mathbf{y} along with the corresponding audio inputs x^{\text{audio}} and prompt \mathbf{p} . The fire and snowflake icons indicate trainable and frozen modules, respectively. The audio decoder is optional. Building on this insight, we introduce DeSTA 2 2 2 DeSTA stands for De scriptive S peech- T ext A lignment, first introduced in [ 8 ] and further improved with self-generated design in [ 9 ] . While the method generalizes beyond speech in this study, we retain the original naming convention for consistency. , a self-generated cross-modal alignment framework that mitigates conflicting supervision by having the backbone language model generate its own training targets. Specifically, we convert the metadata of each audio segment into a structured textual description and pair it with an arbitrary prompt. Then, the LLM generates a corresponding response, which serves as the training target for cross-modal alignment. This self-generated supervision ensures stylistic and semantic consistency with the LLM’s native output distribution, thereby preserving its instruction-following capabilities while enabling effective adaptation to auditory inputs. In our prior work, DeSTA2 [ 9 ] , we demonstrate that training solely on a single general task, speech captioning, is sufficient for learning robust audio grounding. The resulting model exhibited strong zero-shot generalization across

Read the complete text