Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation

Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn, Fengqing Zhu · 2026

Read Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation by Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn, Fengqing Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08967.txt.

Introduction

Siddeshwar Raghavan and colleagues benchmark ongoing audio-video masking without stored examples. Real scenes shift over time, breaking systems trained on static settings, and their benchmark spans four setups across single-source and multi-source datasets. Their ATLAS baseline tunes visual features on audio before cross-modal focus, and Low-Rank Anchoring steadies tuned weights by loss response to curb severe forgetting.

Catalog record

CREATORSiddeshwar Raghavan, Gautham Vinod, Bruce Coburn, Fengqing Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34454
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08967

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn, Fengqing Zhu Abstract Audio-Visual Segmentation (AVS) aims to produce pixel-level masks of sound producing objects in videos, by jointly learning from audio and visual signals. However, real-world environments are inherently dynamic, causing audio and visual distributions to evolve over time, which challenge existing AVS systems that assume static training settings. To address this gap, we introduce the first exemplar-free continual learning benchmark for Audio-Visual Segmentation, comprising four learning protocols across single-source and multi-source AVS datasets. We further propose a strong baseline, ATLAS, which uses audio-guided pre-fusion conditioning to modulate visual feature channels via projected audio context before cross-modal attention. Finally, we mitigate catastrophic forgetting by introducing Low-Rank Anchoring (LRA), which stabilizes adapted weights based on loss sensitivity. Extensive experiments demonstrate competitive performance across diverse continual scenarios, establishing a foundation for lifelong audio-visual perception. Code is available at ∗ 1 1 1 Paper under review - https://gitlab.com/viper-purdue/atlas 1 Introduction Figure 1 : An overview of Exemplar Free Continual Learning Benchmark for Audio-Visual Segmentation Humans can naturally recognize and localize objects by jointly interpreting visual cues and the sounds they produce through everyday experience [ 25 , 34 , 31 ] . This capability is fundamental to how we perceive, understand, and interact with complex environments. Audio-Visual Segmentation (AVS) [ 51 , 50 ] targets this challenge by producing a pixel level map of the objects that produce sounds in the video frames. However, humans continuously adapt to changing surroundings, acquiring new visual and auditory patterns while retaining previously learned knowledge. Hearing a new instrument for the first time, for instance, does not cause listeners to forget the sound of a dog’s bark. Replicating this lifelong learning ability in machines remains highly challenging due to the intricate correspondence and interdependence between audio and visual modalities, the scarcity of precise localization supervision, and the persistent issue of catastrophic forgetting [ 21 , 27 , 20 ] . Since its introduction, AVS has attracted considerable attention, with methods exploring audio-visual correspondence through transformer-based fusion [ 50 ] , bidirectional cross-modal attention [ 15 ] , and temporal audio-visual pixel interaction strategies [ 51 ] . Despite these advances, a critical direction remains under-explored, as real-world audio-visual perception is inherently continual . In practice, a deployed model encounters new sound categories over time, ranging from musical instruments and animals to vehicles and human speech. The model must learn them without retraining from scratch or revisiting prior data. Exemplar Free Continual Learning (EFCL) provides a principled framework for this setting, enabling models to acquire new knowledge sequentially while mitigating catastrophic forgetting [ 21 , 27 , 20 ] without storing any data. The emergence of Pre-Trained Models (PTMs) has further accelerated progress in this direction, as EFCL methods built on strong pre-trained representations [ 43 , 49 , 14 , 16 , 38 ] have demonstrated substantially better real-world applicability compared to their trained-from-scratch frameworks. While Continual Learning (CL) was originally studied in the context of image classification [ 28 , 1 , 48 , 22 , 36 , 37 ] , the field has since expanded considerably, including continual semantic segmentation [ 47 ] , multi-modal continual learning [ 46 ] , and continual reinforcement learning [ 35 ] , reflecting a broader shift toward more realistic and open-ended learning scenarios. However, directly applying these methodologies to the EFCL setting or re-purposing existing methods to the EFCL setting is not trivial. AVS requires the model to simultaneously maintain cross-modal alignment between audio and visual streams, preserve fine-grained spatial segmentation boundaries, and retain previously learned sound producing object associations, all without revisiting or storing past data. The multimodal nature of AVS amplifies forgetting. Degradation in either modality or in their cross-modal alignment can cause failure, even if each modality retains useful information. Beyond forgetting, continual AVS faces several additional challenges. First, the introduction of new sounding categories causes cross-modal interference , where the feature spaces of both the audio and visual encoders must shift to accommodate new patterns, risking misalignment between modalities that were previously well-correlated. Second, the evaluation complexity increases because continual AVS requires dense pixel-level metrics that evaluate segmentation quality and audio-visual alignment while accounting for continual learning across previously encountered tasks. To bridge this gap, we make three contributions. First , we introduce the exemplar-free continual learning benchmark for Audio-Visual Segmentation (CL-AVS), illustrated in Figure 1 . The benchmark evaluates four continual learning protocols across two AVS datasets: Task-Incremental (TIL), Class-Incremental (CIL), and Domain-Incremental Learning (DIL) on the Single Source AVS (SS-AVS) dataset [ 51 ] , and Task-Free Continual Learning [ 2 ] on the Multi-Source AVS (MS-AVS) dataset [ 51 ] . Second , we propose ATLAS , an exemplar-free baseline built on LoRA [ 18 ] adapters for parameter-efficient continual learning. It introduces an audio guided pre-fusion conditioning prior to audio–visual fusion and a Low-Rank Anchoring (LRA) mechanism to mitigate parameter drift across tasks. Third , we conduct extensive experiments across diverse continual learning algorithms and AVS frameworks, with detailed analysis highlighting the challenges of continual AVS and the effectiveness of our baseline 2 Related Works 2.1 Audio-Visual Segmentation Audio-Visual Segmentation (AVS) aims to localize sound producing objects in video frames through pixel-level masks by jointly modeling audio and visual signals. The task was introduced in AVSBench [ 51 ] , which established a benchmark for binary audio-visual segmentation, and later extended to semantic AVS with category-aware predictions [ 50 ] . Early approaches typically adopt encoder–decoder architectures that fuse visual representations with audio embeddings using attention or correlation mechanisms. While effective in controlled environments, these models often struggle in realistic scenes containing multiple sound sources, background noise, and visually salient but silent objects [ 8 ] . Recent AVS research broadly follows two design paradigms. Vision-anchored approaches leverage strong visual priors and treat audio as a conditioning signal, commonly using transformer-based fusion or vision foundation models to improve spatial precision and boundary quality [ 19 , 33 , 26 ] . Although effective, heavy reliance on visual cues may lead to incorrect segmentation when audio–visual correspondence weakens. In contrast, audio-anchored methods emphasize sound-driven localization through bidirectional interaction [ 15 ] or audio-conditioned attention mechanisms [ 6 ] , improving robustness when visual motion is ambiguous but introducing sensitivity to noisy audio. To reduce annotation cost, several works further explore weakly supervised and self-supervised learning by exploiting natural audio–visual correspondence or pseudo-label generation [ 32 , 29 ] . Despite rapid progress, existing AVS methods assume static training distributions where all categories are jointly available. In real-world deployment, however, audio-visual systems encounter continuously evolving environments with new sounding categories appearing over time. To bridge this gap and set up the foundation we propose an exemplar free continual learning benchmark for AVS. 2.2 Continual Learning Continual Learning (CL) studies learning from sequentially arriving data while mitigating catastrophic forgetting [ 21 , 27 , 20 ] . Early work largely focused on image classification, exploring replay-based, regularization-based, and parameter-isolation strategies [ 28 , 22 , 48 , 1 ] . With the emergence of large pretrained models and increasing privacy constraints,

Read the complete text