FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion

Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang, [1.5ex] The Chinese University of Hong Kong, Shenzhen · 2025

Read FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion by Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang, [1.5ex] The Chinese University of Hong Kong, Shenzhen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01111.txt.

Introduction

Shunian Chen and co-authors build a data set of over a million detailed audio captions using models that fuse many cues. Their 2025 paper notes a gap: automated captions for sound are often thin and context poor. Human hearing, by contrast, weaves speech, music, and scene cues together. Their two stage pipeline imitates that. Specialist models first extract cues such as speech, music, general sounds, and video frames.

A large language model then merges those cues into fine grained, context aware captions. The result is a data set of 1.2 million captions plus six million question answer pairs. Models trained on it improved, notably an audio encoder with stronger sound text alignment. The authors present the set, called FusionAudio-1.2M, as a base for richer automated listening. Code and data are public, they note.

Catalog record

CREATORShunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang, [1.5ex] The Chinese University of Hong Kong, Shenzhen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01111

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang, [1.5ex] The Chinese University of Hong Kong, Shenzhen Abstract High-quality, large-scale audio captioning is crucial for advancing audio understanding, yet current automated methods often generate captions that lack fine-grained detail and contextual accuracy, primarily due to their reliance on limited unimodal or superficial multimodal information. Drawing inspiration from human auditory perception, which adeptly integrates cross-modal cues and performs sophisticated auditory scene analysis, we introduce a novel two-stage automated pipeline. This pipeline first employs specialized pretrained models to extract diverse contextual cues (e.g., speech, music, general sounds, and visual information from associated video). A large language model (LLM) then synthesizes these rich, multimodal inputs to generate detailed and context-aware audio captions. Key contributions of this work include: (1) the proposed scalable method for fine-grained audio caption generation; (2) FusionAudio, a new large-scale dataset comprising 1.2 million such detailed captions, combined with 6 million QA pairs; and (3) enhanced audio models developed using FusionAudio, specifically a CLAP-based audio encoder with superior audio-text alignment and instruction following. This paper paves the way for more nuanced and accurate automated understanding of complex audio environments. Code and data can be found in https://github.com/satsuki2486441738/FusionAudio . 1 Introduction The advancement of models like CLAP laionclap2023 for audio retrieval, and GAMA ghosh-etal-2024-gama or Qwen2-Audio Qwen2-Audio for broader audio understanding, heavily relies on large-scale, high-quality audio captioning datasets. Audio captioning has primarily followed two trajectories. Manual annotation drossos2019clothoaudiocaptioningdataset ; kim-etal-2019-audiocaps offers high quality but lacks scalability due to high labor costs. In contrast, automated methods laion-audio-630k ; mei2023wavcaps often use sparse metadata like text labels or tags to assist annotation, while others bai2024audiosetcapsenrichedaudiocaptiondataset ; sun2024autoacdlargescaledatasetaudiolanguage ; yuan2025soundvecapsimprovingaudiogeneration leverage basic multimodal cues. These automated approaches, however, typically rely on limited textual or superficial information, failing to capture rich details (e.g., multimodal contextual details). This results in captions that lack fine-grained details and are prone to hallucinations yang2024airbenchbenchmarkinglargeaudiolanguage , hindering nuanced audio interpretation. Addressing this gap necessitates a paradigm shift. We turn to human auditory perception for inspiration (Figure 1 ). Human auditory understanding leverages sophisticated strategies at two complementary levels. Firstly , humans adeptly integrate cross-modal cues—visual information, for instance, aids speech intelligibility sumby1954visual and sound identification Kayser2010 ; ERNST2004162 . Secondly , auditory scene analysis (ASA) bregman1990auditory allows the auditory system to parse complex soundscapes into distinct streams like speech, music, and ambient sounds based on temporal-spectral regularities Shamma2011 . These sophisticated biological mechanisms offer a compelling blueprint for enhancing automated audio captioning. The impact of this multimodal integration is demonstrated in Table 1 . Current systems, often processing audio in isolation, can misinterpret sounds (e.g., a stationary motorcycle as a moving scooter) or hallucinate details. In contrast, FusionAudio-1.2M leverages comprehensive audiovisual cues to produce more accurate and contextually rich descriptions. Figure 1 : Human auditory perception integrates multisensory cues. Inspired by these principles, we introduce a two-stage pipeline for enhanced automated audio captioning. First, specialized pretrained models extract diverse contextual cues: an Automatic Speech Recognition (ASR) model radford2022robustspeechrecognitionlargescale for speech, a music understanding model zhao2024openmu for musical attributes, an audio understanding model ghosh-etal-2024-gama for general sounds, and a visual model Qwen2.5-VL for video information. Second, a large language model (LLM) qwq32b acts as an integration engine, synthesizing these multimodal cues to generate fine-grained audio captions. This synthesis of rich, cross-modal context by an LLM aims to improve detail and accuracy, addressing prior limitations. Table 1: Comparison of generated captions for a sample audio clip with associated visual context. Hallucinations in prior work are highlighted in red . Improvements from our multimodal approach, FusionAudio, are highlighted in green , demonstrating enhanced accuracy and detail by leveraging visual and comprehensive auditory cues. Method Audio Caption GAMA (Baseline) The audio is dominated by the sound of a motor vehicle engine and intermittent male speech, with wind noise. AudioSetCaps A male and female engage in conversation, their voices audible against a backdrop of ambient noise . The discussion is neutral in tone and does not involve any identifiable objects or language. Auto-ACD A man speaks while a vehicle moves in the distance , possibly on a motor scooter, in an engine room . Sound-VECaps A man is speaking and a vintage motorcycle with a large headlamp, round fuel tank, and sidecar is parked on grass , with the sound of the engine and the man’s voice filling the air, while a vehicle passes by in the background. FusionAudio-1.2M (Ours) Continuous motor vehicle engine noise is prominently featured, accompanied by intermittent male speech with a positive or confirming tone . Wind sounds suggest an outdoor environment , with the engine’s sustained roar maintaining a steady volume throughout the recording. Our contributions are: • Automated fine-grained audio captioning: A pipeline using specialized unimodal models to extract diverse contextual cues, synthesized by an LLM to generate detailed, scalable captions. • FusionAudio-1.2M dataset: A large-scale dataset of 1.2M fine-grained audio captions to advance audio research. • Multimodal cue-enhanced audio models: A CLAP-based audio encoder with improved audio-text alignment, and an instruction-tuned MLLM with stronger audio comprehension and instruction-following. 2 Related Works 2.1 Audio Language Learning The field of audio-language models has seen significant advancements in recent years, with researchers focusing on developing models that can effectively process, understand, and reason about sounds using natural language as a supervision signal. Early works like CLAP Learning Audio Concepts from Natural Language Supervision elizalde2023clap laid the foundation for contrastive learning approaches in audio-language pre-training. Subsequent studies have explored various pre-training objectives, including generative and discriminative methods, to enhance audio representation learning and cross-modal alignment. For instance, CTAL li2021ctal and FLAP yeh2023flap have investigated masked language modeling and cross-attention based masked acoustic modeling for joint representation learning of audio and language modalities. Multi-task learning approaches have also gained attention, with models like UniAudio tian2023uniaudio and SpeechX wang2024speechx demonstrating the potential of unifying diverse audio tasks under a single framework. Furthermore, the integration of large language models (LLMs) with audio processing has opened new avenues for creating more powerful and human-like audio understanding systems. Recent works such as Pengi deshmukh2023pengi , Qwen-audio chu2023qwen , and Audio Flamingo kong2024audio have shown impressive capabilities in handling complex audio tasks through instruction tuning and in-context learning. These advancements highlight the growing importance of audio-language models in bridging the gap between auditory information and language understanding, and their potential for real-world applications. Table 2 : Comparison of open-source audio caption datasets. Name Year # of Audio/QA Avg. Dur (s) Avg. Text Len Visual Music Speech Integration AudioCaps kim-etal-2019-audiocaps 2019 46k/46k 10.00 9.03 ✗ ✗ ✗ ✗ Clotho drossos2019clothoaudiocaptioningdataset 2019 5k/5k 22.50 11.00 ✗ ✗ ✗ ✗ LAION-Audio-630K laion-audio-630k 2022 630k/630k 24.58 7.30 ✗ ✗ ✗ ✗ WavCaps mei2023wavcaps 2024 403k/403k 67.59 7.80 ✗ ✗ ✗ ✗ AudioSetCaps bai2024audiosetcapsenrichedaudiocaptiondataset 2024 1.9M/1.9M N/A 28.00 ✗ ✗ ✗ ✗ Auto-ACD sun2024autoacdlargescaledatasetaudiolanguage 2024 1.5M/1.5M 10.00 18.10 ✓ ✗ ✗ ✓ CompA-R ghosh-etal-2024-gama 2024 62k/200k 9.93 18.00 ✓ ✗ ✗ ✓ FusionAudio-1.2M (Ours) 2025 1.2M/6M 10.00 47.18 ✓ ✓ ✓ ✓ 2.2 Audio Captioning Early audio captioning research relied on manually annotated datasets like AudioCaps kim-etal-2019-audiocaps and Clotho drossos2019clothoaudiocaptioningdataset , which provided high-quality descriptions but

Read the complete text