Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation

Kexin Li, Zongxin Yang *, Yi Yang, Jun Xiao · 2024

Read Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation by Kexin Li, Zongxin Yang *, Yi Yang, Jun Xiao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08161.txt.

Description

Kexin Li and co-authors tackle timing mismatch in audio visual video segmentation in a 2024 paper. The task is to outline the objects making a sound, frame by frame, in step with the audio. Sound gives two clues: which object, and when it starts and stops. Past methods used the first clue but neglected the second, causing timing drift.

Their framework first anchors audio boundaries by using a large language model with retrieval prompts to find points where sound meaning changes. Audio is split into consistent portions at those points. A propagator then inserts audio portion by portion and matches frames as it goes. Tests across three data sets and two backbones improved results, and the method plugs into existing systems. Readers see timing treated as a first class signal.

Catalog record

CREATORKexin Li, Zongxin Yang *, Yi Yang, Jun Xiao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08161

Browse the full library · About the Tower of Babel