SpotSound : Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

Luoyi Sun, Xiao Zhou, Zeqian Li, Ya Zhang, Yanfeng Wang, Weidi Xie † · 2026

Read SpotSound : Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding by Luoyi Sun, Xiao Zhou, Zeqian Li, Ya Zhang, Yanfeng Wang, Weidi Xie † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13023.txt.

Introduction

Sun and five colleagues build SpotSound, an audio-language model that pinpoints exactly when an event occurs inside long recordings. A training objective suppresses invented timestamps for events absent from the audio, while their SpotSound-Bench hides target sounds inside dense background noise, with targets filling under 10 percent of each clip. SpotSound leads temporal grounding benchmarks without losing ground on general audio-language tasks.

Catalog record

CREATORLuoyi Sun, Xiao Zhou, Zeqian Li, Ya Zhang, Yanfeng Wang, Weidi Xie †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58937
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpotSound : Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding Thanks: † Corresponding author. Luoyi Sun, Xiao Zhou, Zeqian Li, Ya Zhang, Yanfeng Wang, Weidi Xie † Abstract . Large Audio-Language Models (ALMs) have recently demonstrated remarkable capabilities in holistic audio understanding, yet they remain unreliable for temporal grounding, i.e. , the task of pinpointing exactly when an event occurs within long-form audio. This limitation stems from two factors: training data dominated by clip-level supervision lacking precise timestamps, and benchmarks that fail to simulate real-world scenarios where short events are obscured by dense background sounds. In this paper, we introduce SpotSound , an audio language model designed for grounding audio events. SpotSound incorporates a novel training objective, specifically designed to suppress hallucinated timestamps for events absent from the input. Additionally, we present SpotSound-Bench , a challenging temporal grounding benchmark where target events occupy less than 10% of each clip, creating a rigorous ‘needle-in-a-haystack’ evaluation. Experiments demonstrate that SpotSound achieves state-of-the-art results on temporal grounding benchmarks while maintaining robust performance across general downstream audio-language tasks. Code, models and benchmark are released on https://loiesun.github.io/spotsound/ 1. Introduction Large Audio-Language Models (ALMs) ( 6 ; 13 ; 9 ) have recently demonstrated remarkable proficiency in holistic tasks, such as generating captions or summarizing entire audio clips. However, they exhibit deficiencies in temporal grounding: the ability to precisely localize specific events within a continuous audio stream. This limitation restricts their deployment in practical applications like security surveillance and media forensics, where precise timing is as crucial as the event classification itself. Figure 1 . Qualitative examples and performance comparison. (a) SpotSound accurately grounds timestamps relevant to the query. (b) SpotSound identifies sound events described in the query that are absent from the audio. (c) Quantitative comparison; blue denotes SpotSound , and red denotes previous top-tier models. We evaluate models on four benchmarks and report the mIoU and [email protected] for each. Generally speaking, two primary factors impede the temporal grounding capabilities of current models: (i) the majority of large ALMs ( 9 ; 1 ; 22 ) are trained on data with coarse, ‘clip-level’ annotations, the models therefore learn to associate acoustic events with the full audio duration rather than precise temporal boundaries; (ii) the community lacks challenging benchmarks to rigorously measure progress for grounding. Existing datasets ( 47 ; 34 ; 11 ) predominantly feature distinct, long-duration sound events separated by silence, targets that are relatively trivial to locate. In contrast, real-world audio is characterized by complex acoustic scenes where short, fleeting events are embedded within continuous background noise, making detection significantly more challenging. Figure 2 . Model architecture and dataset generation pipeline. (a) In SpotSound , we construct an interleaved sequence of timestamps and audio tokens and concatenate with a language query, and feed it to the LLM to predict the target temporal interval. (b) We employ LLMs to generate captions for foreground audio, then randomly mix foreground and background sounds to synthesise the final audio, preserving the insertion timestamp as ground-truth. This paper presents SpotSound , a framework designed to bridge this gap by equipping large ALMs with precise temporal reasoning capabilities. Central to our approach is a temporal encoding mechanism that interleaves timestamp tokens with audio embeddings, enabling accurate event boundary localization for open-vocabulary natural language queries. To ensure response reliability, we introduce a specialized training objective that explicitly targets hallucination mitigation. Specifically, we restructure each training instance into a discriminative quadruplet format, comprising the audio input, a positive query (describing a present sound event), the corresponding ground-truth event timestamps, and a negative query (describing an absent event). This formulation enforces the model to verify acoustic evidence, learning to distinguish genuinely occurring signals from non-existent ones. To support end-to-end training, we curate a temporally-aware audio-language dataset comprising 10k instruction-tuning samples, and collect an additional 67.6k samples from existing audio-language datasets, resulting in 77.6k samples in total. In addition, we address the evaluation gap by introducing a benchmark, termed SpotSound-Bench , which is specifically designed for short-window temporal audio grounding. Unlike prior benchmarks ( 47 ; 34 ; 11 ) that focus on distinct, isolated sound events, SpotSound-Bench embeds short target windows within long-form audio, creating a needle-in-a-haystack scenario. The target events occupy less than 10% of the total duration, forcing the model to pick out fleeting acoustic cues against rich, competing background activity. This makes the benchmark a demanding testbed for evaluating temporal grounding under realistic, dense audio conditions. In summary, we make the following contributions: (i) we endow large ALMs with robust temporal grounding via a training objective that directly suppresses hallucinations on non-existent events; (ii) we incorporate temporal information by interleaving timestamp tokens with audio tokens, giving the model the fine-grained resolution needed to capture the precise timing and duration of acoustic events; (iii) we construct a challenging temporal audio grounding benchmark, SpotSound-Bench , filling a gap in evaluation resources for long-duration, realistic temporal reasoning; (iv) extensive experiments confirm state-of-the-art performance across multiple temporal grounding benchmarks, with the model retaining strong accuracy on the standard sound event detection (SED) task, indicating broad generalization. 2. Methods In this section, we present SpotSound , a framework that endows large Audio-Language Models (ALMs) with the ability of precise temporal grounding across varying audio durations. We start with the problem formulation in Section 2.1 , and detail our architectures and the construction of timestamp-interleaved sequences in Section 2.2 . Lastly, we outline the training paradigm in Section 2.3 . 2.1. Problem Formulation Given an audio stream \mathcal{A}=\{a_{1},\ldots,a_{T}\}\in\mathbb{R}^{1\times T} and a free-form textual query \mathcal{Q} , either a concise label or a descriptive caption, we treat the temporal grounding as a two-stage problem, where the model first answers a binary existence instruction \mathcal{I}_{E} by predicting: \mathcal{B}=\Phi_{\text{SpotSound}}(\mathcal{A},\mathcal{I}_{E}) where \mathcal{B}=\{\mathbf{yes} , \mathbf{no} }. If \mathcal{B}=\mathbf{yes} , the model proceeds to the second stage and answers a grounding instruction, noted as \mathcal{I}_{G} , by localizing all time intervals that semantically match the query: \mathcal{W}=\Phi_{\text{SpotSound}}(\mathcal{A},\mathcal{I}_{G}) where \mathcal{W}=\{(s_{1},e_{1}),\ldots,(s_{K},e_{K})\} correspond semantically to the query, where each pair (s_{k},e_{k}) represents a respective start and end timestamp. 2.2. Audio Temporal Grounding Model In this section, we provide details of the proposed SpotSound model for audio temporal grounding that localizes time spans in an audio recording corresponding to a natural-language query. Large Audio Language Backbone. Contemporary large ALMs typically couple an audio encoder with a large language model (LLM). In this study, we adopt the two representative models, Qwen2-Audio ( 6 ) and Audio Flamingo 3 ( 13 ) , as our backbone models. Both of these backbones initialize Whisper-large-v3 ( 38 ) as audio encoder. In this process, raw audio \mathcal{A} is initially resampled to 16kHz and subsequently converted into 128-channel mel-spectrograms \mathcal{M}=\{m_{1},\ldots,m_{T}\}\in\mathbb{R}^{T_{\text{mel}}\times F} using a 25ms window and a 10ms hop length, where T_{\text{mel}} and F represent the time and frequency channels. The spectrograms are subsequently encoded into audio tokens via the audio encoder, \mathbf{A}_{i}=\phi_{\text{audio}}(m_{i}) . The encoder incorporates a pooling layer with a stride of two, compressing the temporal length of the audio representation. Each output timestep of the encoder corresponds to approximately 40ms of the original audio signal. Qwen2-Audio ( 6 ) establishes a strong baseline by leveraging Qwen2-7B ( 49 ) as its language model, while Audio Flamingo 3

Read the complete text