Multimodal Alignment with Cross-Attentive GRUs for Fine-Grained Video Understanding
NAMHO KIM, JUN-HWA KIM · 2025
Read Multimodal Alignment with Cross-Attentive GRUs for Fine-Grained Video Understanding by NAMHO KIM, JUN-HWA KIM (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03531.txt.
Introduction
Namho Kim and Junhwa Kim build a system that reads video, still frames, and text together, in work from twenty twenty-five. Fine-grained video tasks need both motion cues and meaning cues at once. A single stream of data often misses part of that mix. Their system joins the three streams so each can fill gaps in the others.
GRU units read each stream in order, and cross-attention lets the streams query each other. Extra training tricks add small shifts to features and rebuild them to aid learning. The team tests the system on two hard tasks. One spots real-world violence in video. The other rates how pleasant and how strong a face looks. In both tests, the joined system beats single-stream baselines by a clear margin.
Catalog record
| CREATOR | NAMHO KIM, JUN-HWA KIM |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03531
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Alignment with Cross-Attentive GRUs for Fine-Grained Video Understanding NAMHO KIM, JUN-HWA KIM Abstract Fine-grained video classification requires understanding complex spatio-temporal and semantic cues that often exceed the capacity of a single modality. In this paper, we propose a multimodal framework that fuses video, image, and text representations using GRU-based sequence encoders and cross-modal attention mechanisms. The model is trained using a combination of classification or regression loss, depending on the task, and is further regularized through feature-level augmentation and autoencoding techniques. To evaluate the generality of our framework, we conduct experiments on two challenging benchmarks: the DVD dataset for real-world violence detection and the Aff-Wild2 dataset for valence-arousal estimation. Our results demonstrate that the proposed fusion strategy significantly outperforms unimodal baselines, with cross-attention and feature augmentation contributing notably to robustness and performance. K eywords Multi-Modal \cdot LLM \cdot Clip \cdot Violence Detection \cdot Valance-Arousal Estimation 1 Introduction This study aims to enhance the intricate scene analysis capabilities required for demanding video understanding tasks such as violence detection and affective state estimation. While vision-based architectures, including convolutional neural networks (CNNs) and transformer models, have achieved significant progress in visual recognition, they often fall short in capturing subtle semantic distinctions, particularly in fine-grained or emotionally rich scenarios. These limitations are further exacerbated in video settings, where spatio-temporal dependencies must be jointly modeled across multiple frames. To mitigate such limitations, prior research has explored the integration of multimodal information—including vision, text, and audio—to provide complementary cues for comprehensive scene understanding. For instance, approaches like CLIP align visual and textual embeddings to boost classification or retrieval accuracy in joint vision-language tasks [ 1 ] . Similarly, challenges in affective behavior analysis, such as ABAW [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ] , have demonstrated the utility of combining facial expressions, audio, and contextual information for robust emotion recognition [ 9 , 2 ] . Despite these advancements, many existing models frequently rely on complex transformer-based fusion mechanisms, which, while effective, often incur high computational costs and complexity during both training and inference. In this paper, we propose a robust and efficient multimodal fusion framework that incorporates three parallel modalities: video segments capturing motion dynamics, sampled image frames offering rich spatial detail, and textual captions extracted from keyframes that convey explicit semantic information. Each modality is processed through a frozen pre-trained encoder—specifically, a 3D CNN for video, a vision transformer for images, and a language encoder for text—followed by GRU-based sequence encoders to model temporal features within each respective stream. We employ a bidirectional cross-attention mechanism to intricately integrate the modality-specific features, facilitating dynamic and context-aware interaction among video, image, and text representations. The resulting fused embedding is subsequently passed through a shallow multilayer perceptron for final classification or regression. Model training is guided by a composite loss function: cross-entropy is utilized for classification tasks (e.g., violence detection on the DVD dataset), and mean squared error is applied for regression tasks (e.g., valence-arousal estimation on the Aff-Wild2 dataset). To validate our approach, we conduct comprehensive experiments on two challenging benchmark datasets: the DVD dataset for real-world violence detection [ 18 ] , and the Aff-Wild2 dataset for continuous valence-arousal prediction [ 15 ] . Both datasets present demanding scenarios that explicitly require robust multimodal reasoning. Our experimental results consistently demonstrate that the proposed framework significantly outperforms unimodal and naive fusion baselines. Furthermore, thorough ablation studies confirm that both the cross-attention mechanism and feature-level augmentation are critical components, significantly enhancing the model’s robustness and performance. This research contributes a practical and highly effective architecture for fine-grained video understanding and paves the way for broader applications in critical domains such as public safety, healthcare, and advanced human-computer interaction. 2 Related Work 2.1 Multimodal Feature Representation Recent advancements in affective behavior analysis and scene understanding have emphasized the importance of multimodal feature extraction from video data. In large-scale benchmarks such as the Aff-Wild2 dataset, visual, audio, and textual information are often jointly exploited to improve recognition accuracy across diverse affective tasks [ 9 ] . Rather than relying solely on raw video input, researchers have increasingly adopted pretrained encoders to obtain more robust and semantically meaningful features from each modality. For visual representation, deep convolutional and transformer-based models such as ResNet [ 19 ] , EfficientNet [ 20 ] , and masked autoencoders (MAE) [ 21 ] have been widely used, often trained on domain-specific datasets like facial expression corpora to improve generalization. In parallel, audio features are commonly extracted using pretrained models such as VGGish [ 22 ] or Wav2Vec2 [ 23 ] , which capture both low-level acoustic properties and high-level emotional tones. Textual features, derived from transcriptions or generated captions, are typically embedded using large language models such as BERT [ 24 ] . Multimodal setups have also expanded to incorporate three or more streams. Unlike earlier approaches that focused solely on vision or audio, recent research increasingly combines all three—visual, audio, and language—to capture the richness of human behavior and scene semantics. These multimodal embeddings form the foundation for downstream tasks including emotion regression and violence detection. In our work, we adopt a similar philosophy by encoding video segments, image frames, and generated captions using frozen pretrained models. Unlike many prior approaches that focus on facial regions or speech alone, we combine scene-level visual features with semantic textual information to enrich the understanding of complex situations such as aggression or emotional dynamics. 2.2 Multimodal Fusion Strategies Beyond feature extraction, how these multimodal representations are fused plays a crucial role in determining system performance. Early works commonly employed naive fusion strategies, such as direct concatenation followed by fully connected layers. However, such methods often struggle to model inter-modal relationships effectively. Recent approaches increasingly adopt attention-based fusion mechanisms. Transformer-based models [ 25 ] use self-attention to jointly process features from multiple modalities, learning complex dependencies. While powerful, these models are often computationally intensive and require large datasets to train effectively. For temporal modeling, GRUs [ 26 ] and Temporal Convolutional Networks (TCNs) [ 27 ] offer efficient alternatives, particularly when working with smaller datasets or deploying on resource-constrained systems. Our approach builds on these trends by employing GRU-based sequence encoders for each modality and integrating them via bidirectional cross-attention. This design enables each modality to inform and enhance the others while maintaining modularity and computational efficiency. By doing so, we bridge the semantic and temporal gaps between modalities without the overhead of full transformer-based fusion. 3 Methodology Our proposed multimodal framework is designed to integrate video, image, and text modalities derived from a single video source. As shown in Figure 1 , three modality-specific streams are sampled and processed in parallel. Each stream is passed through a pretrained encoder followed by a GRU-based sequence encoder, and the resulting embeddings are fused using cross-attention mechanisms before being passed to the final prediction head. To ensure temporal coverage and semantic diversity, we uniformly sample 64 segments from each input video. From these, a subset of 16 or 4 samples is chosen per modality stream depending on the encoder type and data