ACAVCaps: Enabling Large-Scale Training for Fine-Grained and Diverse Audio Understanding

Yadong Niu 1 , Tianzi Wang 1,2 , Heinrich Dinkel 1 , Xingwei Sun 1 , Jiahao Zhou 1 , Gang Li 1 , Jizhong Liu 1 , Junbo Zhang 1 , Jian Luan 1 · 2026

Read ACAVCaps: Enabling Large-Scale Training for Fine-Grained and Diverse Audio Understanding by Yadong Niu 1 , Tianzi Wang 1,2 , Heinrich Dinkel 1 , Xingwei Sun 1 , Jiahao Zhou 1 , Gang Li 1 , Jizhong Liu 1 , Junbo Zhang 1 , Jian Luan 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24038.txt.

Introduction

Niu and co-authors introduce ACAVCaps, a large dataset of detailed audio captions for training audio-language models. Past caption sets lack the scale and fine detail such models need. ACAVCaps draws from the ACAV100M collection. A multi-expert pipeline studies each clip for speech, music, and acoustic traits. A large language model then merges those views into rich captions. Models pretrained on ACAVCaps generalize better on downstream tasks than models trained on other leading caption sets, they report. The dataset is publicly released.

Catalog record

CREATORYadong Niu 1 , Tianzi Wang 1,2 , Heinrich Dinkel 1 , Xingwei Sun 1 , Jiahao Zhou 1 , Gang Li 1 , Jizhong Liu 1 , Junbo Zhang 1 , Jian Luan 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17235
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ACAVCaps: Enabling Large-Scale Training for Fine-Grained and Diverse Audio Understanding Yadong Niu 1 , Tianzi Wang 1,2 , Heinrich Dinkel 1 , Xingwei Sun 1 , Jiahao Zhou 1 , Gang Li 1 , Jizhong Liu 1 , Junbo Zhang 1 , Jian Luan 1 Abstract General audio understanding is a fundamental goal for large audio-language models, with audio captioning serving as a cornerstone task for their development. However, progress in this domain is hindered by existing datasets, which lack the scale and descriptive granularity required to train truly versatile models. To address this gap, we introduce ACAVCaps, a new large-scale, fine-grained, and multi-faceted audio captioning dataset. Derived from the ACAV100M collection, ACAVCaps is constructed using a multi-expert pipeline that analyzes audio from diverse perspectives—including speech, music, and acoustic properties—which are then synthesized into rich, detailed descriptions by a large language model. Experimental results demonstrate that models pre-trained on ACAVCaps exhibit substantially stronger generalization capabilities on various downstream tasks compared to those trained on other leading captioning datasets. The dataset is avaiable at https://github.com/xiaomi-research/acavcaps. † † address: 1 MiLM Plus, Xiaomi Inc, Beijing, China 2 The Chinese University of Hong Kong, Hong Kong, China 1 Introduction The ability to comprehend complex acoustic environments is a fundamental goal in the pursuit of general artificial intelligence. Large Audio-Language Models (LALMs) have recently emerged as a promising paradigm for this task, aiming to build models with a deep and versatile understanding of sound. A cornerstone for developing such models is the task of audio captioning—generating rich, human-readable descriptions of acoustic content, which serves as a powerful bridge between the audio and text modalities. Despite their rapid progress, the performance of current LALMs on diverse, real-world audio understanding tasks remains constrained. We argue that this limitation stems not from the models themselves, but from the data they are trained on. Existing audio captioning datasets suffer from several critical issues: (I) Scarcity of high-Fidelity data at scale. Creating large-scale datasets with accurate and detailed descriptions is inherently challenging. Manual annotation is costly and difficult to scale; (II) Homogeneous Sources and Rigid Annotations. Most large-scale datasets are confined to limited domains (e.g., AudioSet) and characterized by a single, stylistic pattern that lacks the linguistic variety found in the real world; (III) Lack of Descriptive Granularity. Captions are often too generic (e.g., “a man is speaking”) and fail to provide the discriminative acoustic features needed to distinguish between nuanced auditory events. Effective audio-text alignment is crucial for LALMs, which requires a training dataset that provides a fine-grained mapping from audio to descriptive text. High-quality, detailed captions enable this, allowing models to learn generalizable representations for diverse tasks. We introduce ACAVCaps, a large-scale, fine-grained audio captioning dataset derived from ACAV100M [ 16 ] . To create detailed captions, we analyze audio with specialized expert models and synthesize their outputs using a Chain-of-Thought (CoT) enhanced large language model (LLM). This pipeline produces richly detailed descriptions to train more capable audio models. Figure 1: Data construction and evaluation frameowrk. Table 1: Comparison of ACAVCaps with existed caption datasets. The Unique Tokens column reports the total number of unique tokens within each dataset, as counted by the Qwen3 tokenizer. \dagger MP-LLM: Multiple Experts Models and LLM; \ddagger Multi-Domain: This includes speech, music and sound-events ( \diamond denotes that domain were not elaborated in detail); \lx@sectionsign Extended Multi-Domain: This includes speech, music, sound-events, combinations thereof, and silence. Labeling Dataset Duration (h) Samples Unique Tokens Domain Source Manual AudioCaps [ 15 ] 135 50k 5.5k Multi-Domain \ddagger , \diamond AudioSet Clotho [ 9 ] 24 3.8k 5.5k Multi-Domain \ddagger , \diamond FreeSound SongDescriber [ 17 ] 12 0.4k 2.4k Music MTG-Jamendo LLM MusicCaps [ 1 ] 7 4.6k 4.6k Music AudioSet LPMusicCaps [ 8 ] 127 21.6k 5.1k Music Audioset, MSD WavCaps [ 19 ] 1.8k 0.4M 23.1k Multi-Domain \ddagger , \diamond AudioSet Auto-ACD [ 22 ] 5.2k 1.9M 20.3k Multi-Domain \ddagger , \diamond AudioSet Sound-VeCaps [ 24 ] 4.5k 1.6M 42.7k Multi-Domain \ddagger , \diamond AudioSet AudioSetCaps [ 3 ] 5.6k 2.0M 20.9k Multi-Domain \ddagger , \diamond AudioSet MP-LLM \dagger ACAVCaps (Ours) 13.0k 4.7M 76.7k Extended Multi-Domain \lx@sectionsign ACAV100M 2 Related works The advancement of general audio understanding is intrinsically linked to the quality and diversity of training datasets. The field has evolved from smaller, manually annotated corpora to large-scale, automatically generated ones, yet significant challenges related to data scale, descriptive granularity, and source limitations persist across all major audio domains. In the domain of general sound events, early foundational datasets like AudioCaps [ 15 ] and Clotho [ 9 ] were created through intensive manual annotation. While providing high-quality human descriptions, their data scale is inherently limited (typically a few thousand samples), and their captions often lack fine-grained detail, focusing on generic, event-level descriptions. To address the scale issue, a new wave of datasets was created using automated pipelines. These include WavCaps [ 19 ] , AudioSetCaps [ 3 ] , and Auto-ACD [ 22 ] , which scaled up to millions of audio-caption pairs. However, this increase in scale came at the cost of descriptive quality. The data sources for these automated methods are often a limiting factor; for instance, WavCaps refines web-crawled text which is often coarse, while Auto-ACD and Sound-VECaps [ 24 ] rely on paired video data, restricting their applicability to audio-only contexts. Consequently, despite their size, these datasets often fail to resolve the core problem of descriptive granularity. Table 2: Performance of audio captioning models pre-trained on various datasets, evaluated on the MECAT-Caption benchmark. The final score is a weighted average of three main categories: systematic, Content-Specific, and Content-Unrelated. For all metrics, a higher score indicates better performance. Notably, a ’Pure’ sample contains only one content type (e.g., only speech), while a ’Mixed’ sample contains a combination of two or three types (i.e., speech, music, and sound events). \dagger Combined refers to the combination of AudioSetCaps, Auto-ACD, WavCaps, and Sound-VECaps. Training Dataset Systematic Content-Related Content-Unrelated Score Long Short Speech Music Sound Environment Pure Mixed Pure Mixed Pure Mixed AudioSetCaps [ 3 ] 52.4 52.0 30.2 31.4 44.3 30.9 52.4 21.6 15.4 37.4 Auto-ACD [ 22 ] 47.3 50.0 29.1 31.0 26.9 21.9 49.5 18.9 11.0 32.8 WavCaps [ 19 ] 47.3 50.9 27.3 30.1 15.9 19.4 46.5 20.0 9.2 31.4 Sound-VeCaps [ 24 ] 47.0 49.7 29.1 30.3 27.2 21.9 49.8 18.7 11.4 32.8 Combined \dagger 52.2 54.1 30.2 32.2 45.4 23.3 52.7 20.2 11.1 36.6 ACAVCaps 76.6 75.7 64.2 64.9 60.5 41.1 59.5 28.0 34.8 60.9 The music domain shows a similar trajectory. Manually annotated datasets like MusicCaps [ 1 ] and SongDescriber [ 18 ] offer rich, detailed captions but are limited in scale. In response, automatically labeled datasets such as LPMusicCaps [ 8 ] have emerged, leveraging LLMs to generate captions from existing musical databases. While larger, their descriptive granularity is often constrained by the richness of the source metadata, which may not contain the nuanced details of instrumentation, mood, and texture that a deep acoustic analysis could provide. For speech, large-scale datasets have historically focused on tasks like automatic speech recognition (ASR), with their annotations capturing lexical content (transcripts) rather than descriptive captions of the acoustic scene. This leaves a significant gap,

Read the complete text