Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

Wenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo · 2025

Read Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning by Wenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19938.txt.

Introduction

Wenrui Li and five co-authors split sight and sound into two streams so a spiking network can recognize actions it never trained on. Zero-shot audio-visual learning fails when background scenes bias the guess or motion detail washes out. Their dual-stream transformer peels context away from sparse motion. One unit learns the scene jointly across senses, while images converted to events capture movement sharply.

A third block models how sound itself moves. Neuron firing thresholds flex with global motion and context, steadying the spiking net. On standard benchmarks the design beat prior bests, with accuracy gains of 26.2% and 39.9% on key measures.

Catalog record

CREATORWenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning Thanks: This work was supported in part by the National Key R&D Program of China (2023YFA1008500), the National Natural Science Foundation of China (NSFC) under grants 624B2049, 62402138, and the Fundamental Research Funds for the Central Universities under grants HIT.DZJJ.2024025. (Corresponding author: Xiaopeng Fan.) Thanks: Wenrui Li and Wangmeng Zuo are with the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China. (e-mail: [email protected]; [email protected]). Thanks: Penghong Wang, Xingtao Wang and Xiaopeng Fan are with the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China, and also with Harbin Institute of Technology Suzhou Research Institute, Suzhou 215104, China. (e-mail: [email protected]; [email protected]; [email protected]). Thanks: Yonghong Tian is with the School of AI for Science, the Shenzhen Graduate School, Peking University, Shenzhen, China, the Peng Cheng Laboratory, Shenzhen, China and also with the School of Computer Science, Peking University, Beijing, China (e-mail: [email protected]). Wenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo Abstract Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2% and 39.9%. I Introduction Zero-Shot learning (ZSL) tasks in the audio-visual domain aim to use both audio and visual modalities to classify target classes that have not been observed during training. Supervised audio-visual methods [ 1 , 2 , 3 , 4 ] train models with input data and corresponding labels, allowing them to predict new inputs. However, previous methods are limited in recognizing previously unseen classes, restricting them to categorizing only the classes in the training data. Training models for every possible data class in real-world settings is impractical. Therefore, generalized zero-shot learning extends ZSL by aiming to classify both unseen and seen classes during testing. Currently, most of the existing audio-visual ZSL methods [ 5 , 6 , 7 ] have focused on projecting temporal and semantic information accurately to generate more precise audio-visual representations. However, these methods mainly focus on effectively modeling and integrating temporal and semantic information, overlooking two critical factors: background scene bias and motion information . Previous research [ 8 , 9 ] has demonstrated that specific action categories in mainstream video datasets are closely related to the background scenes where the actions occur. When models overemphasize contextual semantics, they will degrade by primarily encoding fine-grained scene details. For instance, a model might misclassify a video as “Running” simply because a running track is presented, even if the actual activity is “Baby Sleeping”. This issue conflicts with the fundamental goal of video representation learning introduces background scene bias. This is a significant concern when applied to diverse datasets. Decoupling motion information from video inputs to optimize multimodal feature learning is still challenging. Fig. 1: To mitigate background bias and highlight the differences among similar types of videos, we converted RGB images into events. This conversion only occurs when there are significant changes in the background scene. To address these challenges, we propose an Event Generation Model (EGM) to convert RGB images into events, eliminating background scene bias and capturing essential motion details. As shown in Fig. 1 , our EGM effectively removes background noise from images in scenarios such as “Baseball” and “Basketball Dunk”. Moreover, EGM captures and highlights critical motion elements, such as the movements of players and objects. For action categories with similar semantic contexts, such as “Basketball” and “Basketball Dunk”, our method effectively distinguish them by capturing distinct motion information, including audience reactions, player movements, and ball positions. Event data, characterized by the time and location of events, exhibits spatial and temporal sparsity. We utilize Spiking Neural Networks (SNNs) to process sparse event data, which naturally encode temporal information and directly extract features without additional preprocessing or dimensionality reduction. Moreover, the memory capabilities of SNNs make them well-suited for attention mechanisms. When utilizing SNNs to process video data with long contextual dependencies, it is crucial to ensure the following two aspects: 1) Developing Spiking Attention Mechanisms : Implement spiking attention to enhance the long-term dependencies of spiking features. This integration enables SNNs to effectively capture and interpret complex temporal dynamics in the data, significantly improving performance in tasks requiring long-range dependency understanding. 2) Effectively Utilizing Timestep Information : Wisely use timestep information to reduce spiking redundancy. By employing a multi-stage timestep shrinkage strategy, the SNN maintains efficiency in feature extraction while minimizing inference latency. This approach ensures each network layer captures important features at its specific temporal resolution, optimizing the accuracy and efficiency of processing complex audio-visual data. This paper aims to separate contextual semantic information from dynamic motion data to minimize background scene bias and enhance the focus on motion for video classification. We use a Recurrent Joint Learning Unit (RJLU) to integrate latent information from various modalities, promoting efficient joint learning. The RJLU helps the network understand contextual semantics and the environments where actions occur. By combining the Event Generation Model (EGM) with Spiking Neural Networks (SNNs), our model effectively captures both motion and temporal information. To tackle the challenge of capturing motion cues from audio, we introduce a Discrepancy analysis block (DAB) that models the differences between converted visual events and extracted audio information, defining these differences as audio motion cues. To enhance the robustness of SNNs in capturing dynamic temporal and motion information, we also dynamically adjust the thresholds of Leaky Integrate-and-Fire (LIF) neurons based on statistical cues from global motion and contextual semantics. To better capture long-range dependencies in SNNs, MDST++ combines SNNs with self-attention mechanisms to further enhance the model’s learning capability. To fully exploit multi-scale temporal information, we propose a multi-stage timestep shrinkage strategy that ensures information retention by gradually decreasing time steps. These advancements make MDST++ particularly effective in handling complex sequential and event data patterns, resulting in superior classification performance. The main contributions could be summarised as follows: • We introduce an innovative dual-stream architecture MDST to decouple background scene and motion information effectively. This is the first work that utilizes synthetic events with SNNs for audio-visual ZSL. • We utilize the EGM and SNNs to capture sparse motion information and mitigate background scene bias. By dynamically adjusting the threshold of spiking neurons, we further enhanced the robustness and efficiency of SNNs in processing temporal and motion information. • We propose the RJLU for extracting contextual semantic

Read the complete text