Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation
Kexin Li, Zongxin Yang *, Yi Yang, Jun Xiao · 2024
Read Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation by Kexin Li, Zongxin Yang *, Yi Yang, Jun Xiao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08161.txt.
Description
Kexin Li and co-authors tackle timing mismatch in audio visual video segmentation in a 2024 paper. The task is to outline the objects making a sound, frame by frame, in step with the audio. Sound gives two clues: which object, and when it starts and stops. Past methods used the first clue but neglected the second, causing timing drift.
Their framework first anchors audio boundaries by using a large language model with retrieval prompts to find points where sound meaning changes. Audio is split into consistent portions at those points. A propagator then inserts audio portion by portion and matches frames as it goes. Tests across three data sets and two backbones improved results, and the method plugs into existing systems. Readers see timing treated as a first class signal.
Catalog record
| CREATOR | Kexin Li, Zongxin Yang *, Yi Yang, Jun Xiao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08161