Mi DashengLM: Efficient Audio Understanding with General Audio Captions
Horizon Team, MiLM Plus · 2025
Read Mi DashengLM: Efficient Audio Understanding with General Audio Captions by Horizon Team, MiLM Plus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03983.txt.
Introduction
Heinrich Dinkel and co-authors present an open model that understands audio of all kinds, trained on plain-language captions. Large audio language models often learn from closed data or other private models, which limits sharing and reuse. Their 2025 system, MiDashengLM, learns from general audio captions describing speech, music, and everyday sound together. A new training set, ACAVCaps, supplies those captions at scale. The model aims to be both efficient and broad.
Caption training teaches the model to say what it hears, a flexible skill that transfers to questions and sorting tasks. Openness is a design goal: other teams should be able to inspect, run, and extend the system. The authors position the model as shared infrastructure for audio AI. Efficiency keeps it usable outside giant labs.
Catalog record
| CREATOR | Horizon Team, MiLM Plus |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mi DashengLM: Efficient Audio Understanding with General Audio Captions Horizon Team, MiLM Plus Abstract Current approaches for large audio language models (LALMs) often rely on closed data sources or proprietary models, limiting their generalization and accessibility. This paper introduces MiDashengLM, a novel open audio-language model designed for efficient and comprehensive audio understanding through the use of general audio captions using our novel ACAVCaps training dataset. MiDashengLM exclusively relies on publicly available pretraining and supervised fine-tuning (SFT) datasets, ensuring full transparency and reproducibility. At its core, MiDashengLM integrates Dasheng, an open-source audio encoder, specifically engineered to process diverse auditory information effectively. Unlike previous works primarily focused on Automatic Speech Recognition (ASR) based audio-text alignment, our strategy centers on general audio captions, fusing speech, sound and music information into one textual representation, enabling a holistic textual representation of complex audio scenes. Lastly, MiDashengLM provides an up to 4 × \times speedup in terms of time-to-first-token (TTFT) and up to 20 × \times higher throughput than comparable models. Checkpoints are available at and . 1 Introduction Large language models (LLMs) have played a pivotal role in advancing machine learning approaches for natural language processing (NLP), demonstrating impressive capabilities in understanding the world through text. While these models can effectively interact with humans via text, the ability to understand sound remains crucial for agents to fully engage with the physical world. Large Audio-Language Models (LALMs) aim to bridge the gap between auditory and textual understanding. Within the audio domain, we identify three commonly used broad categories: speech, (environmental) sounds and music. Aligning audio with text requires a mapping between speech/sound/music and respective text. For speech the most common alignment are transcripts, while captions are used for sound and music. Transcripts can be understood as a monotonous alignment between audio and text domains. In contrast, captions are typically used for broader audio elements like sounds and music, offering a more generalized alignment, meaning they capture the overall nature or occurrence of a sound. Current audio understanding research typically processes speech transcripts, audio captions, and music captions separately. This independent approach limits the depth and completeness of auditory scene analysis. Anther key limitation stems from existing audio captions, which often offer only superficial descriptions. For example, spoken content is frequently simplified to “somebody is speaking”, ignoring semantic details. Furthermore, these datasets often fail to capture critical auditory aspects like room acoustics (e.g., reverberation) or signal quality. To overcome these limitations, this paper proposes fusing speech transcripts, audio captions, and music captions into a single, unified general caption. Our goal is to create a holistic textual representation that jointly includes all relevant audio information, providing a more detailed and semantically rich description of the auditory environment. 1.1 Motivation Developing a LALM requires aligning audio features with textual descriptions. Utilizing sound and music captions as a training target has been previously explored [ 1 ; 2 ; 3 ; 4 ] to enhance audio understanding. However, these approaches lack automatic speech recognition (ASR) capabilities, limiting their usefulness for general applications, as users expect a LALM to handle both general audio understanding and speech — not just captions. The most used alignment paradigm couples large language models (LLMs) with audio understanding through automatic speech recognition (ASR). This approach prevails for two key reasons: First, numerous high-quality off-the-shelf ASR models exist that can generate reasonably accurate transcripts automatically. Second, a substantial portion of internet audio content consists of speech-based material - including podcasts, lectures, interviews, and other spoken-word formats - making ASR an effective bridge between audio and text modalities. Several prominent works have demonstrated the effectiveness of ASR-based LALM training, such as Whisper [ 5 ] , SpeechT5 [ 6 ] , Universal Speech Model (USM) [ 7 ] , Open Whisper-style Model (OWSM) [ 8 ] and Kimi-Audio [ 9 ] . However, we argue that ASR-based pretraining provides limited benefits for general audio-language understanding, due to the following reasons: Inefficient Data Utilization Large-scale pretraining on million-hour long datasets typically relies on existing automated speech recognition (ASR) pipelines to generate transcripts from speech. This results in a substantial loss of potentially valuable data, as sounds like music, environmental noises, or even silent pauses are discarded. Using a general captioning approach has the benefit that any audio can be used for training, as even “noisy” audio clips could be labeled. This significantly enhances data diversity, allowing models to learn from a much wider range of acoustic information beyond just speech. Trivial objective The training losses for ASR-based LALMs are typically low, even across different languages, suggesting that the models learn relatively little meaningful information from ASR-based data, compared to text-based training [ 10 ] (see Figure 1 ). We attribute this to the simplicity of speech-text alignments, where the temporal ordering of acoustic units and their corresponding text tokens follows a monotonic (left-to-right) correspondence. Thus a model only needs to establish local correspondences between spoken words and their textual counterparts, bypassing the need to understand broader (global) audio context. Limitations of ASR-Based Pretraining Beyond Speech Content ASR-based pretraining does not focus on information other than the spoken content. This limited scope means that important speech meta-information, such as a speaker’s gender, age, or emotional state, is not captured or integrated during the pretraining process. Furthermore, the pretraining methodology overlooks audio signal-specific characteristics like reverberation levels, recording quality, and environmental acoustics. 1.2 Audio caption and speech summarization Audio captions have been the focus of extensive research [ 11 ; 12 ; 13 ] . Most datasets during the start of the audio-caption era were manually labeled [ 14 ; 11 ; 12 ; 15 ] , but recent work has leveraged large language models (LLMs) to scale and streamline dataset creation. Notable LLM-assisted audio captioning datasets include WavCaps [ 16 ] , AutoACD [ 17 ] , SoundVECaps [ 18 ] , AudiosetCaps [ 19 ] and FusionAudio-1.2M [ 20 ] . These works utilized LLMs in order to enhance existing audio captions by additional visual information [ 18 ] , temporal information [ 17 ] or with additional CLAP filtering [ 19 ; 3 ] . However, we identify two key limitations in existing datasets: Neglect of spoken language: Publicly available captioning data primarily focuses on sound/music events and their audio-visual/temporal relationships, despite speech constituting the majority of real-world audio [ 21 ] . Current audio captioning datasets can therefore be better understood as (environmental-) sound captioning datasets. Limited data diversity: Popular datasets (AudioCaps, WavCaps, AutoACD, SoundVECaps, AudiosetCaps and FusionAudio-1.2M) predominantly derive from the same audio sources (Audioset [ 21 ] , VGGSound [ 22 ] and FSD50k [ 23 ] ). This source overlap leads to a problematic one-to-many mapping: multiple “distinct” datasets are, in fact, derived from identical underlying audio clips, containing different textual descriptions. This redundancy adds little training audio data variation, limiting model generalization. While audio captions have been used for LALM pretraining, existing approaches typically generate new captions through either (1) paraphrasing existing descriptions [ 3 ; 24 ] or (2) augmenting them with (unrelated) video context [ 4 ] using LLMs, rather than genuinely diversifying the underlying audio content. In our work we rely on general audio captions , a novel captioning type.