AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Yaoting Wang, Yun Zhou, Zipei Zhang, Henghui Ding · 2026

Read AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes by Yaoting Wang, Yun Zhou, Zipei Zhang, Henghui Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02724.txt.

Introduction

Wang and co-authors build a dataset for tracking active speakers in complex real scenes. Audio-visual speaker tracking locates who is speaking. It uses sound and vision together. Video editing, surveillance, and human-computer interaction all need it. Existing datasets use simple scenes with coarse annotations. Evaluation then rewards static audio-visual matching. Robust modeling of dynamic scenes goes untested.

Their dataset, AVTrack, targets human-centric audio-visual instance segmentation. Scenes include camera motion, occlusions, and position changes. Conditions are diverse and hard by design. Representative methods lose substantial performance on it. That drop establishes AVTrack as a tough benchmark. A simple yet effective baseline is also provided. The authors aim to push audio-visual scene understanding toward real-world complexity.

Catalog record

CREATORYaoting Wang, Yun Zhou, Zipei Zhang, Henghui Ding
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02724

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes Yaoting Wang, Yun Zhou, Zipei Zhang, Henghui Ding Abstract Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding. This capability is essential for real-world applications such as intelligent video editing, surveillance, and human–computer interaction. However, existing datasets are largely limited to simple or homogeneous audio-visual scenes with coarse annotations. Such oversimplified settings bias evaluation toward static audio–visual co-occurrence, rather than rigorously assessing robust spatiotemporal modeling and cross-modal reasoning in complex, dynamic scenes. To address these limitations, we introduce AVTrack , a human-centric audio-visual instance segmentation (AVIS) dataset designed for dynamic real-world scenarios. AVTrack features diverse and challenging conditions, including camera motion, visual occlusions, and position changes. Evaluations of representative AVIS methods on AVTrack reveal substantial performance degradation, establishing AVTrack as a challenging benchmark for robust human-centric audio-visual scene understanding in complex environments. We further provide a simple yet effective baseline to facilitate future research. Project website: https://FudanCVL.github.io/AVTrack/ . Figure 1 : Illustrative samples from our proposed AVTrack benchmark. Audio signals are omitted for visual clarity. AVTrack features challenging human-centric audio-visual scenarios, such as instance scale dynamics, visual occlusion, camera motion change, and relative position change. In contrast, previous datasets are typically dominated by simple settings, such as static cameras and single-instance scenes. A more detailed description is provided in Section 3 . Zoom in to inspect the instances in extremely small sizes. 1 Introduction Humans naturally perceive and interpret their surroundings by seamlessly integrating visual and auditory signals. This cross-modal capability allows us to effortlessly identify who is speaking in a group, track a clapping individual amid visual clutter, or follow human activities under occlusion and rapid motion. Inspired by this perceptual ability, prior work ( 29 ; 65 ; 30 ) has explored audio-visual speaker tracking , which leverages synchronized audio and visual streams to localize sounding individuals while maintaining identity consistency over time. As a core component of multimodal perception, audio-visual tracking underpins a wide range of applications, including human–computer interaction, augmented reality, security surveillance, and intelligent video editing. Beyond speaker localization, recent progress in audio-visual segmentation (AVS) ( 43 ; 69 ; 51 ; 19 ) has significantly advanced multimodal scene understanding by enabling pixel-level delineation of sounding objects. Among these efforts, audio-visual instance segmentation (AVIS) ( 19 ) represents a critical step forward, aiming to simultaneously detect, segment, and track sounding instances across video frames at the object level. By integrating instance-level reasoning with audio-visual correspondence, AVIS holds substantial promise for robust perception in complex, real-world settings, particularly in human-centric scenes. Building reliable audio-visual correspondence is central to effective fine-grained multimodal learning. Early works such as AVS-Bench ( 69 ; 68 ) establish end-to-end CNN-based frameworks for jointly modeling visual and auditory cues. To better capture long-range temporal dependencies, AVSegFormer ( 14 ) introduces transformer-based architectures that significantly enhance spatiotemporal audio-visual reasoning. More recently, GAVS ( 49 ) demonstrates strong data efficiency by integrating audio-visual correlation adapters into the visual foundation model SAM ( 25 ) via adapter tuning. In parallel, COMBO ( 59 ) advances cross-modal fusion by explicitly modeling bilateral audio-visual relations. Building upon these advances, AVISM ( 19 ) combines Mask2Former ( 7 ) and VITA ( 22 ) extending audio-visual understanding to the instance level with window-based attention to achieve memory-efficient AVIS. Despite notable progress, existing tasks and benchmarks still fall short in realistically evaluating AVIS systems. Early audio-visual speaker tracking datasets ( 26 ; 38 ; 39 ) are mainly collected in controlled laboratory settings, with few speakers and largely static scenes. Although AVA-ActiveSpeaker ( 42 ) increases speaker diversity, it is restricted to a single data source (TV series) and provides only bounding-box annotations without cross-frames identity consistency. Meanwhile, AVS benchmarks ( 69 ; 68 ) offer pixel-level labels in in-the-wild scenarios, but are dominated by short, trimmed clips (5–10 seconds), limiting long-range temporal modeling. AVISeg ( 19 ) partially addresses this issue by extending clip duration to around 60 seconds for instance-level segmentation; however, most scenes remain visually simple, with limited camera motion, background variation, and relative position shifts, particularly in human-centric scenarios. Such oversimplified settings fail to capture human-centric real-world challenges such as frequent occlusions, multi-person interactions, and complex spatiotemporal dynamics in the human-centric applications. To address these challenges, we introduce AVTrack , a novel audio-visual benchmark designed for human-centric AVIS in complex scenes. AVTrack includes 871 video clips for 3,120 densely annotated instance tracklets, spanning 8 challenging conditions such as Camera Motion Change, Visual Occlusions, and Multiple Instances, which significantly complicate the modeling of cross-modal spatiotemporal dynamics. We benchmark state-of-the-art Video Instance Segmentation (VIS) and AVIS methods on AVTrack and observe significant performance degradation compared to existing datasets. These findings highlight AVTrack as a more realistic and challenging testbed for evaluating the robustness and advancements of human-centric AVIS systems. Additionally, we introduce a simple yet effective baseline framework to further support the community’s research in fine-grained and human-centric audio-visual understanding. In summary, our main contributions are as follows: • We introduce AVTrack for benchmarking human-centric AVIS in complex and dynamic scenes. • Experiments on AVTrack reveal significant performance gaps for VIS and AVIS methods, and we present a simple yet effective baseline with modular, plug-and-play extensibility for future research. • We perform an in-depth analysis of the challenges posed by AVTrack and discuss promising directions for advancing human-centric AVIS in complex scenes. 2 Related Works 2.1 Audio-Visual Speaker Tracking Audio-visual speaker tracking aims to track active speakers. Prior work has primarily relied on a small set of controlled datasets that limit real-world complexity. The most commonly used classic corporas such as AV16.3 ( 26 ) and CAV3D ( 38 ) provide multi-speaker audio-visual recordings in indoor meeting settings, but consist of recordings with controlled camera and microphone configurations, constraining diversity in scene dynamics and speaker variation. The AVRI ( 39 ) dataset increases recording duration, but remains constrained in environment and annotation scope. These benchmarks are valuable for evaluating tracking algorithms but do not capture the complexity of unconstrained real-world audio-visual scenarios beyond laboratory conditions. 2.2 Audio-Visual Segmentation AVS ( 69 ) aims to locate and segment sounding objects in a video, serving as a key step toward multimodal scene understanding. The pioneering work AVSBench ( 69 ) establishes the foundational benchmark with audio-visual object segmentation for single sound sources (AVS-SS) and for multiple sound sources (AVS-MS), which focus on producing binary masks corresponding to active audio emitters. Building upon this, audio-visual semantic segmentation (AVSS) ( 68 ) further enriches the task by introducing semantic labeling, requiring the model to predict category-specific segmentation maps aligned with sound cues. Ref-AVS ( 51 ) further advances multimodal grounding by introducing the reference audio-visual segmentation (Ref-AVS) task, which leverages textual cues describing visual and auditory information to guide flexible, context-aware segmentation. Recently, AVISeg ( 19 ) extends the audio-visual domain to AVIS, integrating detection, instance-level mask generation, and temporal association across frames. AVS is challenging due to complex cross-modal correspondence and multi-source interference. Existing AVS datasets, however, mostly contain simple and static scenes, limiting their ability to reflect real-world audio-visual complexity. 2.3 Complex Scene Understanding Complex scene understanding has long been

Read the complete text