DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction
Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi · 2025
Read DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction by Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10070.txt.
Introduction
Kiana Hooshanfar and co-authors present a 2025 model that predicts where people will look in a video, using sound as well as sight. Sound cues draw the eye, yet most models read frames alone. Adding sound well is hard and costly. Their model fuses sight and sound tokens in a shifting way that keeps cost down while catching long-range links.
A sight branch weighs key tokens and merges them across scales. A sound branch reads raw audio for cues. A fusion block blends the two through local, global, and adaptive streams. Layered decoders then draw the final attention map. Tests on six standard video sets back the design. Such maps serve video coding, ads, and tools that guide the eye.
Catalog record
| CREATOR | Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi Abstract Audio-visual saliency prediction aims to mimic human visual attention by identifying salient regions in videos through the integration of both visual and auditory information. Although visual-only approaches have significantly advanced, effectively incorporating auditory cues remains challenging due to complex spatio–temporal interactions and high computational demands. To address these challenges, we propose D ynamic T oken F usion Sal iency (DFTSal), a novel audio-visual saliency prediction framework designed to balance accuracy with computational efficiency. Our approach features a multi-scale visual encoder equipped with two novel modules: the Learnable Token Enhancement Block (LTEB), which adaptively weights tokens to emphasize crucial saliency cues, and the Dynamic Learnable Token Fusion Block (DLTFB), which employs a shifting operation to reorganize and merge features, effectively capturing long-range dependencies and detailed spatial information. In parallel, an audio branch processes raw audio signals to extract meaningful auditory features. Both visual and audio features are integrated using our Adaptive Multimodal Fusion Block (AMFB), which employs local, global, and adaptive fusion streams for precise cross-modal fusion. The resulting fused features are processed by a hierarchical multi-decoder structure, producing accurate saliency maps. Extensive evaluations on six audio-visual benchmarks demonstrate that DFTSal achieves SOTA performance while maintaining computational efficiency. 1 Introduction Human vision naturally identifies salient regions in dynamic scenes, guiding attention and enabling efficient information processing [ 29 ] . In computer vision, saliency prediction models attempt to replicate this human capability by generating saliency maps that highlight where observers are most likely to focus [ 48 ] . These models are essential in a wide range of applications, including marketing [ 20 , 26 ] , multimedia [ 41 ] , object detection [ 2 , 67 ] , image/video compression [ 17 ] , user-interface design [ 26 ] , and cognitive research [ 12 ] . Although significant progress has been made for static images [ 21 , 1 ] , predicting saliency in videos remains challenging due to the need to capture multi-scale spatial-temporal cues and handle complex motion. Recent video saliency methods have employed deep learning architectures such as CNN-LSTM frameworks and transformer-based networks to improve performance [ 54 , 58 , 23 , 8 , 38 , 66 ] . At the same time, incorporating auditory information has proven beneficial for capturing salient events in complex scenarios [ 69 ] , as demonstrated by DAVE [ 50 ] and STAViS [ 51 ] . Meanwhile, models such as CASP-Net [ 61 ] , DiffSal [ 62 ] and RAVF [ 63 ] integrate audio-visual cues through various cross-modal fusion strategies, but often suffer from large-scale networks, complex attention mechanisms, or incomplete spatio–temporal modeling [ 22 ] . These limitations underscore a persistent challenge, achieving high predictive accuracy while maintaining computational efficiency and robust cross-modal alignment, particularly when dealing with dynamic audio-visual data in resource-constrained scenarios. To address these challenges, we propose a novel audio-visual saliency framework, D ynamic T oken F usion Sal iency Prediction (DTFSAL). Our approach leverages a multi-scale visual encoder to capture detailed spatial-temporal features from videos, while a dedicated audio branch processes audio signals. We then introduce three key modules to refine and fuse these representations: Learnable Token Enhancement Block (LTEB) : Adaptively emphasizes crucial spatio-temporal cues by weighting salient tokens and suppressing less relevant features. Dynamic Learnable Token Fusion Block (DLTFB) : Employs a shifting mechanism that reorganizes and merges tokens, improving long-range dependencies and preserving important spatial details. Adaptive Multimodal Fusion Block (AMFB) : Integrates audio and visual features via three complementary streams (local, global, and adaptive), enabling precise cross-modal alignment. Finally, a hierarchical multi-decoder generates multi-scale saliency maps, which are combined through a final 3D convolution to yield a coherent saliency prediction. In summary, our main contributions are: • We propose DTFSal, a novel audio-visual saliency model that efficiently fuses multi-scale video and audio features for improved performance. • We introduce three specialized modules (LTEB, DLTFB, and AMFB) that jointly refine spatio-temporal features, capture long-range dependencies, and adaptively fuse cross-modal information. • We demonstrate state-of-the-art (SOTA) results on challenging audio-visual saliency benchmarks while preserving computational efficiency, showing that our approach achieves a balance between accuracy and resource demands. Figure 1 : Overview of our DTFSal model, which integrates a multi-scale encoder, a hierarchical multi-decoder, LTEB, DLTFB, and AMFB for efficient and accurate audio-visual saliency prediction. 2 Related Work Visual Saliency Prediction: Early video saliency prediction methods relied on motion-based models and manual feature extraction but struggled to capture spatio-temporal relationships, limiting their effectiveness in tracking object motions over time. With the rise of deep learning, these traditional approaches have been largely replaced by more advanced models capable of learning complex spatial and temporal patterns directly from video data. Early approaches extended image-based models with temporal modules, such as ACLNet [ 55 ] , an attentive CNN-LSTM that extracts frame-wise features and processes them with a ConvLSTM enhanced by a supervised spatial attention module trained on large static saliency datasets. Other comparable models utilized two-stream architectures that separately processed appearance and motion while incorporating an objectness stream to capture salient objects, enhancing performance at the cost of increased model size [ 22 , 28 , 65 ] . These CNN+RNN and multi-stream networks showcased the effectiveness of deep spatio-temporal feature fusion for saliency prediction but often struggled to capture long-range dependencies in complex scenes [ 39 , 64 ] . More recent methods [ 61 , 66 , 57 , 43 ] have increasingly adopted Transformer-based models to overcome these limitations. Transformers, using self-attention mechanisms, effectively capture long-range and multi-scale temporal dependencies in video data. For instance, Wang et al. [ 61 ] integrate self-attention into a 3D CNN backbone to model global spatial-temporal contexts, while other works, such as TMFI-Net [ 66 ] , Combine multi-scale temporal features through Transformer-based feature integration to achieve SOTA accuracy, though at the cost of large model sizes [ 51 ] . While Transformer-enhanced methods deliver remarkable performance, they highlight a trade-off between capturing global context and ensuring computational efficiency. UNISAL and TinyHD offers a more lightweight alternative; however, it falls short in accuracy and robustness compared to newer architectures that achieve a more optimized balance [ 13 , 22 , 63 ] . Audio-Visual Saliency Prediction: Incorporating audio cues into video saliency models has become a growing focus due to the significant influence of sound on visual attention [ 68 , 63 , 59 ] . The DAVE model [ 50 ] illustrates this approach, introducing the Audio-Visual Eye-tracking dataset (AVE) and a simple encoder-decoder that integrates audio and visual features to enhance saliency prediction [ 50 ] . Subsequently, STAViS, a spatio-temporal audio-visual saliency network, demonstrated that synchronized audio, such as speech and noise events, enhances saliency prediction in dynamic scenes [ 51 ] . More specialized models [ 35 , 46 ] have been developed to focus on multi-speaker conversation videos, predicting saliency and localizing sound sources. Advancements like CASP-Net [ 61 ] and DiffSal [ 62 ] further improve performance through enhanced fusion strategies and increased model complexities, scaling up to 76M parameters. A newer methodology, DAVS [ 68