DDAVS: Disentangled Audio Semantics and Delayed Bidirectional Alignment for Audio-Visual Segmentation

Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang, Isaac Ning Lee, Xulong Bai, Tianrui Zhu, Jingxuan Niu, Yansong Tang · 2025

Read DDAVS: Disentangled Audio Semantics and Delayed Bidirectional Alignment for Audio-Visual Segmentation by Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang, Isaac Ning Lee, Xulong Bai, Tianrui Zhu, Jingxuan Niu, Yansong Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20117.txt.

Description

Jingqi Tian leads an eight-author team teaching machines to find what is making a sound in a video. Audio-visual segmentation must localize sound-producing objects at pixel level, but multiple overlapping sources get entangled, and audio-visual misalignment lets the loudest or largest object dominate while subtler sources vanish. Their framework, DDAVS, attacks both failure modes explicitly.

Learnable queries extract audio semantics and anchor them in a structured space built from an audio prototype memory bank, with contrastive learning sharpening discrimination between sources. Dual cross-attention with deliberately delayed modality interaction then steadies the alignment between what is heard and what is seen. Across the AVS-Objects and VPO benchmarks, spanning single-source, multi-source, and multi-class settings, the method reached state-of-the-art performance.

Catalog record

CREATORJingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang, Isaac Ning Lee, Xulong Bai, Tianrui Zhu, Jingxuan Niu, Yansong Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20117

Browse the full library · About the Tower of Babel