CA 2 ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition

Jongseo Lee 1 , Joohyun Chang 1 , Dongho Lee, Jinwoo Choi 2 · 2025

Read CA 2 ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition by Jongseo Lee 1 , Joohyun Chang 1 , Dongho Lee, Jinwoo Choi 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23447.txt.

Description

Naming the action in a video takes both a sense of space and a sense of time, and most models lean too hard on one. Four authors build a transformer system with separate spatial and temporal experts that swap information at every layer through a bottleneck cross-attention step. Each side's judgment is thus informed by the other, while the exchange stays cheap.

A fuller version adds a third expert for sound, letting hearing inform seeing. Tested on standard action benchmarks, the vision-only system performed in a balanced way across datasets that stress different skills. The audio-visual version also did well on four benchmarks that include sound. The authors present the design as a route to video recognition that treats sight and sound as partners, and they show the cross-attention module is where the benefit is created.

Catalog record

CREATORJongseo Lee 1 , Joohyun Chang 1 , Dongho Lee, Jinwoo Choi 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23447

Browse the full library · About the Tower of Babel