AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound
Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier · 2025
Read AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound by Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14142.txt.
Introduction
Gijs Wijngaard and co-authors present a model that reasons about the meaning of sounds, in work from 2025. Models that link sound and language can label sounds well, but they often miss fine details of what a sound means. The team builds their model around a framework of sound meaning drawn from how humans hear and sort sounds.
To train and test the model, they built a new dataset with sound clips paired with careful captions, cleaned to avoid overlap with test data used elsewhere. Their model beat leading rivals in several training setups. Both the model and the dataset are released in public. The work aims to help machines describe and reason about sound in richer, more human terms.
Catalog record
| CREATOR | Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14142
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier Abstract Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AudSemThinker , a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AudSem , a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AudSem addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AudSemThinker outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AudSemThinker and the AudSem dataset are released publicly 1 1 1 https://github.com/GLJS/AudSemThinker . 1 Introduction Recent foundation models, such as OpenAI’s o1 [ 49 ] and DeepSeek R1 [ 25 ] , have shown that incorporating explicit reasoning phases before generating answers significantly improves performance across tasks. Despite these advances, audio-language models have rarely implemented such structured reasoning approaches. Although general foundation models increasingly separate generation into explicit “thinking” and “answering” stages, audio understanding models still rely largely on end-to-end architectures that directly map audio into text without intermediate reasoning steps. This gap represents a promising opportunity to improve audio understanding capabilities through more deliberate reasoning mechanisms. Several techniques have emerged to extend test-time reasoning in a controlled and scalable manner. One such method is budget forcing, which inserts “wait” tokens during generation to lengthen the model’s reasoning process [ 47 ] . Other approaches rely on reinforcement learning to manage a model’s thinking budget—the computational effort allocated during inference. For instance, Length-Controlled Policy Optimization (LCPO) [ 2 ] jointly optimizes for accuracy and reasoning length, while cosine-shaped reward functions have been used to penalize overly long yet incorrect outputs [ 11 ] . Notably, scaling test-time compute has shown significant benefits, with smaller models like LLaMA 3B outperforming much larger ones, such as LLaMA 70B, when allowed more extensive reasoning during inference [ 4 ] . Figure 1: Schematic overview of the AudSemThinker model and the AudSem dataset . Left: The AudSemThinker model’s reasoning process, involving a thinking phase, semantic descriptor analysis, and answer generation. Right: Example of four types of tasks in the AudSem dataset: open-ended question answering, multiple-choice question answering, audio captioning, and creative writing. A parallel challenge in audio-language modeling is the limited diversity of training data. The field has become highly compartmentalized, with most models trained on overlapping datasets primarily sourced from AudioSet [ 20 ] and Freesound [ 17 ] . This redundancy is reflected in widely used benchmarks such as AudioCaps [ 33 ] , WavCaps [ 45 ] , and various derivatives. For example, studies report that 17.6% of AudioCaps overlap with WavCaps, while Clotho [ 16 ] exhibits even higher overlaps - up to 89% - with other datasets [ 37 ] . Such cross-contamination contributes to a homogenized dataset landscape, limiting the generalization and robustness of audio-language models [ 63 ] . To address the dual challenges of limited reasoning in audio-language models and the lack of diverse training data, we introduce AudSemThinker (Audio Semantics Thinker), a model that grounds its reasoning process in a structured framework of auditory semantics inspired by human cognition. Specifically, the model analyzes audio in terms of functional components, including sound-generating agents (who), physical sound sources (what), generation mechanisms (how), and contextual cues (when/where). This framework draws on theoretical foundations [ 18 ] and experimental research [ 26 , 5 ] that investigate the perceptual and cognitive mechanisms driving human perception of complex sounds [ 23 ] . To support the development of AudSemThinker , we also construct AudSem , a new dataset synthesized from YouTube closed captions, designed to enhance training diversity and reduce overlap with existing resources. Using the AudSem dataset, we train AudSemThinker using two paradigms, Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO), and evaluate its performance on a range of established benchmarks, spanning both general audio foundation tasks and audio-based dialogue. In summary, our main contributions are: • AudSemThinker — a novel audio-language model that integrates structured reasoning with semantic descriptor analysis, achieving state-of-the-art performance across multiple tasks. • AudSem — a newly curated dataset built from YouTube closed captions, specifically designed to reduce overlap with existing datasets. Sound descriptions in AudSem are generated via a multi-stage synthetic pipeline, ensuring both diversity and high quality. 2 Related Work 2.1 Audio Datasets with Synthetically Generated Captions Early audio-language datasets primarily relied on manual annotation or existing metadata. For example, AudioCaps [ 33 ] employed Amazon Mechanical Turk workers to generate captions for AudioSet clips. Subsequently, the SAM-S dataset [ 28 ] introduced an innovative approach by extracting sound-specific data from closed caption transcripts of Hollywood movies, yielding 116,000 captions. A significant shift occurred with the advent of AI-assisted dataset creation for audio models. In the audio domain, 80% of new datasets created in the first half of 2024 incorporated LLM-generated captions, a marked increase from 69.2% in 2023 [ 63 ] . One notable example is Auto-ACD [ 57 ] , which employed multiple AI models to process both audio and visual information from VGGSound and AudioSet. The dataset creation pipeline integrated seven different models, including BLIP-2 [ 39 ] for image captioning and the CLIP model [ 52 ] for categorization. Other examples of AI-assisted dataset creation for audio models include Sound-VECaps [ 68 ] , which employed CogVLM [ 60 ] for visual captioning and LLaMA 3 [ 42 ] for generating audio descriptions. This approach produced two complementary dataset variants: Sound-VECaps F , which incorporates detailed visual features, and Sound-VECaps A , which focuses solely on audible content. Building on this, AudioSetCaps [ 3 ] introduced a more refined three-stage pipeline that combines large audio-language models for content extraction, LLMs for caption generation, and CLAP-based refinement [ 64 ] , resulting in 1.9 million high-quality audio-caption pairs. Our AudSem dataset creation methodology advances beyond prior work by integrating audio, video, and YouTube closed caption data through a novel pipeline. This approach employs an ensemble of specialized AI models for multimodal analysis. Table 1 compares our pipeline with other synthetic audio caption datasets. Model Video Audio Text Image No AS Think Step Text Input Num Models Length WavCaps [ 45 ] ✗ ✓ ✓ ✗ ✗ ✗ Metadata 1 44.87 AudioSetCaps [ 3 ] ✗ ✓ ✓ ✗ ✗ ✗ Labels 4 179.20 AF-AudioSet [ 35 ] ✗ ✓ ✗ ✗ ✗ ✗ - 1 143.57 Auto-ACD [ 57 ] ✗ ✓ ✓ ✓ ✗ ✗ Labels 7 103.87 Sound-VECaps F [ 68 ] ✓ ✓ ✓ ✗ ✗ ✗ Labels 3 180.72 AudSem (ours) ✓ ✓ ✓ ✓ ✓ ✓ CC 9 852.63 Table 1: Comparison of synthetic audio caption dataset creation pipelines. The AudSem pipeline uses all four modalities (video, audio, text and image) for its data across nine models. Instead of AudioSet, closed captions were used in its creation. The AudSem dataset contains thinking and semantic descriptor steps, and