Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning

Hossein Rajoli, Jie Ji, Xiaolong Ma, Fatemeh Afghah · 2025

Read Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning by Hossein Rajoli, Jie Ji, Xiaolong Ma, Fatemeh Afghah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24919.txt.

Introduction

Hossein R Nowdeh, Jie Ji, Xiaolong Ma, and Fatemeh Afghah tame a snag in joint learning from more than one sense: one sense can drown the rest. Their frame, M-SAM, works with many sense types and with both early and late ways to join them. It acts in three steps in each pass of training.

First it finds the top sense by a fair share score of its part in the wins. Next it reshapes the loss to force sturdiness in that top sense first. Last it moves the weights with the reshaped slopes. That way the weak senses still add their own traits. On four sets, the frame beat the newest train and slope tricks.

Catalog record

CREATORHossein Rajoli, Jie Ji, Xiaolong Ma, Fatemeh Afghah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning Hossein Rajoli, Jie Ji, Xiaolong Ma, Fatemeh Afghah Abstract In multimodal learning, dominant modalities often overshadow others, limiting generalization. We propose Modality-Aware Sharpness-Aware Minimization (M-SAM), a model-agnostic framework that applies to many modalities and supports early and late fusion scenarios. In every iteration, M-SAM in three steps optimizes learning. First, it identifies the dominant modality based on modalities’ contribution in the accuracy using Shapley. Second, it decomposes the loss landscape , or in another language, it modulates the loss to prioritize the robustness of the model in favor of the dominant modality, and third, M-SAM updates the weights by backpropagation of modulated gradients. This ensures robust learning for the dominant modality while enhancing contributions from others, allowing the model to explore and exploit complementary features that strengthen overall performance. Extensive experiments on four diverse datasets show that M-SAM outperforms the latest state-of-the-art optimization and gradient manipulation methods and significantly balances and improves multimodal learning. 1 Introduction Multimodal learning [ 27 ] has become the cornerstone of emerging deep neural models. These advanced models integrate diverse data types, including text, audio, images, and sensor data, mimicking the multiplicity of sensory information that humans experience. (a) AGM (b) SAM (c) M-SAM Figure 1 : Loss landscape visualization of CREMA-D (late fusion) for AGM, SAM, and M-SAM from two different viewpoints. This capability for integration is crucial for a range of complex tasks. This integration capability is crucial for a range of complex tasks, enabling models to fully grasp context through multiple viewpoints, essential in a wide spectrum of applications such as autonomous driving, human-computer interaction, and intelligent virtual agents. By blending various sources of information, these multi-modal approaches can construct richer and more robust representations, potentially leading to improved performance across various domains. However, achieving the ideal balance among modalities presents a unique challenge, as one modality can dominate the learning process, limiting the model’s ability to develop unique discriminative capabilities from less dominant data sources. In practice, the imbalanced modality can lead to unexpected results, where the performance of jointly trained models shows minimal improvement or even declines compared to unimodal models. [ 4 , 33 ] . Some studies have found that different modalities often converge at varying speeds, a phenomenon known as uncoordinated convergence [ 23 , 28 , 30 , 34 , 16 ] . Modality encoders sometimes overfit due to the nature of the input data and the underlying task, especially when overexposed to their training data. [ 23 , 28 , 6 ] . This overexposure causes the model to adhere too closely to the noise and unique characteristics of the training data, resulting in poor generalization to new, unseen data. Similarly, the shared network, which integrates features from all modalities, can be overfitted to the combined feature space, reducing its ability to generalize effectively. This uncoordinated convergence can significantly hinder the performance of multi-modal models. While many researchers [ 28 , 23 ] argue that multi-modal networks fail to fully utilize individual modalities if their performance drops in mono-modality evaluations compared to single-modality training, such claims may overlook key aspects of how these networks are structured. For instance, late fusion techniques [ 41 , 38 , 8 ] , where modalities are kept separate until the final layers and allow each modality’s impact to be more easily assessed. In contrast, early fusion merges modalities from the start and process them together. In these cases, the contributions of individual modalities become intertwined, making their individual effects less distinguishable but not necessarily less significant. The fact that a modality does not perform as well individually within a complex multi-modal system does not mean it is not providing valuable information. Evaluating these networks requires a more profound look rather than a straightforward comparison of isolated performances. (a) Audio (b) Visual (c) Multi-modal Figure 2 : The normalized overfitting gap ( \tau ) comparisons of joint-trained, OGM-GE, AGM, SAM, and our proposed M-SAM on the AV-MNIST dataset. Viewing this in color is recommended. Recent SOTA [ 23 , 28 ] studies, which utilize gradient modulation techniques, tailor backpropagation for each modality, aiming to find minima that are perfectly suited to each modality. While this approach does outperform previous modulation-based works by achieving better accuracy on both modalities and overall, huge gap shows it tends to overfit [ 18 ] , leading to sharper minima [ 12 ] that constrain the model’s ability to explore shared and complementary high semantic cues across modalities. The degree of overfitting is evaluated using the normalized overfitting gap, \tau , measuring the discrepancy between a model’s training and test performance. It is defined as: \mathcal{\tau}=\frac{\left|Acc_{train}-Acc_{test}\right|}{Acc_{test}}, (1) with 0\leq\tau\leq 1 , and assuming models are not underfitted, values closer to 0 signify robust generalization, whereas values nearing 1 indicate pronounced overfitting. Such a normalized overfitting gap has been established as a proxy for sharpness and generalization ability [ 18 , 7 ] , and it clearly illustrates the advantage of our method in learning flatter, generalizable minima. Figure 2 shows that throughout training, our M-SAM method maintains a close match between training and validation accuracy, indicating a well-balanced learning process. Figure 1 further highlights differences in curvature by visualizing the loss landscapes. The relationship between flatter, wider minima and reduced overfitting gap is clearly reflected in M-SAM. By relaxing the strict perturbation constraint used in traditional SAM, M-SAM finds broader, flatter minima. This outcome aligns with the findings of Friendly-SAM [ 24 ] that argues traditional SAM not necessarily gets to the flattest minima. In our paper, we present Modality-Aware SAM (M-SAM), a method designed to boost network generalization by giving precedence to the dominant modality. By steering the learning process towards flatter minima that favor the dominant modality, M-SAM enhances its resilience against parameter updates from other modalities during backpropagation. Additionally, this approach allows non-dominant modalities the freedom to explore and search for optimal features aligned with overall performance. Unlike recent studies that have only slightly improved encoder performance through narrow-focused gradient modulation techniques, M-SAM actively searches for broader, flatter areas in the loss landscape that benefit all modalities and the shared network. This approach prevents our system from becoming overly fine-tuned to the particular characteristics of the training data. The proposed method also dynamically adjusts the learning process by continuously evaluating and rebalancing the contribution of each modality in favor of the dominant modality’s generalization in each iteration that not only enhances the weaker modalities to search for their optima but also ensures a more balanced, robust learning across the board. As our main contributions, we show that: • Our method is conceptually aligned with findings that dominant modalities can emerge during multi-modal training due to dynamics such as modality competition [ 15 ] . We propose a practical optimization-based solution that dynamically adapts to such imbalances during training. Specifically, we introduce Modality-Aware SAM (M-SAM), which detects and prioritizes the dominant modality at each iteration and mini-batch, ensuring its robustness against noisy gradient contributions from weaker modalities and promoting more stable multi-modal convergence. • M-SAM with a modality-focused generalization capability enhances the resilience and stability of the dominant modality against weight updates from

Read the complete text