BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates
Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le, Cam-Van Thi Nguyen · 2026
Read BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates by Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le, Cam-Van Thi Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19718.txt.
Introduction
Nguyen and co-authors tackle lopsided learning when data comes from several modes at once. In real settings each mode can be missing at a different rate. Strong modes then swamp the weak ones during training. Their BALM framework plugs into existing models without changing their design. One module recalibrates single-mode features with global context. A second module rebalances gradients across modes in size and direction. Tests on emotion recognition benchmarks showed steady gains under missing data.
Catalog record
| CREATOR | Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le, Cam-Van Thi Nguyen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78125 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19718
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le, Cam-Van Thi Nguyen Abstract Learning from multiple modalities often suffers from imbalance, where information-rich modalities dominate optimization while weaker or partially missing modalities contribute less. This imbalance becomes severe in realistic settings with imbalanced missing rates (IMR), where each modality is absent with different probabilities, distorting representation learning and gradient dynamics. We revisit this issue from a training-process perspective and propose BALM , a model-agnostic plug-in framework to achieve balanced multimodal learning under IMR. The framework comprises two complementary modules: the Feature Calibration Module ( FCM ), which recalibrates unimodal features using global context to establish a shared representation basis across heterogeneous missing patterns; the Gradient Rebalancing Module ( GRM ), which balances learning dynamics across modalities by modulating gradient magnitudes and directions from both distributional and spatial perspectives. BALM can be seamlessly integrated into diverse backbones, including multimodal emotion recognition (MER) models, without altering their architectures. Experimental results across multiple MER benchmarks confirm that BALM consistently enhances robustness and improves performance under diverse missing and imbalance settings. Code available at: https://github.com/np4s/BALM_CVPR2026.git 1 Introduction Multimodal learning has achieved substantial progress by jointly exploiting complementary information from audio, visual, and textual modalities 2 ; 31 ; 20 ; 47 . However, real-world multimodal systems rarely operate under perfect conditions, as sensor failures, recording noise, or acquisition costs often cause partial or complete modality loss 46 ; 54 ; 30 . This phenomenon leads to incomplete multimodal learning , where missing data affect both representation quality and cross-modal interaction 54 . Effectively handling incomplete or missing modalities remains a central challenge for building reliable multimodal systems 21 . Beyond incompleteness, multimodal learning inherently faces modality imbalance 45 ; 49 ; 38 , where modalities contribute unequally due to differences in signal reliability, feature granularity, and data scale. When modalities are missing, this imbalance worsens as the dominant modality overfits while the less frequent or corrupted ones receive limited supervision, leading to a biased equilibrium that weakens robustness and generalization. Figure 1 : Modality-specific missing rates’ distribution of a tri-modal dataset under Share Missing Rate (SMR=0.5). Figure 2 : Model accuracy under SMR=0.5 and five equivalent IMR settings with (r_{A},r_{L},r_{V})\in\{0.3,0.5,0.7\} on IEMOCAP. Recent advances in missing-modality learning have explored two main paradigms to enhance robustness under incomplete data: alignment-based methods , which align representations between complete and incomplete inputs using contrastive, correlation, or augmentation techniques 22 ; 30 ; 19 ; 27 , and generation-based methods , which reconstruct missing modalities through autoencoders, variational inference, graph reasoning, or diffusion models 26 ; 56 ; 40 ; 50 . These approaches generally overlook the imbalance that arises when missingness occurs during training. Notably, some models 19 ; 7 compute reconstruction losses using the full data, whereas some approaches 56 ; 50 do not employ reconstruction but rely on pretrained encoders trained on full modalities, leaving challenges of incomplete training data unaddressed. Most prior studies further simplify the problem by adopting the Shared Missing Rate (SMR) assumption 19 ; 7 ; 56 , where all modalities are randomly dropped with the same probability. As illustrated in Fig. 1 , when SMR\!=\!0.5 , each modality exhibits comparable missing proportions, resulting in a statistically balanced yet idealized setting. In contrast, realistic multimodal systems often encounter uneven data degradation across modalities due to heterogeneous acquisition or noise. We refer to this scenario as the Imbalanced Missing Rate (IMR) condition 33 ; 34 ; 55 , which introduces persistent exposure disparity and optimization bias across modalities. As shown in Fig. 2 , traditional missing-modality models exhibit unstable behavior under IMR settings, particularly when the missing rates across modalities differ significantly. This imbalance causes performance fluctuations, as models struggle to adapt when one modality becomes substantially less reliable than the others. Our approach. We study multimodal learning under the Imbalanced Missing Rate (IMR) setting, where modality absence occurs from the beginning of training and follows distinct missing rates. Imbalanced Missing Rate introduces two coupled challenges: (1) representation imbalance , where heterogeneous missing patterns distort unimodal feature distributions and hinder consistent cross-modal fusion 36 ; and (2) learning imbalance , where gradients are dominated by frequently observed modalities, leading to biased convergence 8 . To address these issues, we propose BALM ( B alanced A gnostic L earning under Imbalanced M issing Rates ), a lightweight and model-agnostic plug-in framework that mitigates both representation- and optimization-level imbalance under IMR. Our propose BALM integrates seamlessly with existing multimodal backbones, particularly those used in MER. It comprises two complementary modules: a Feature Calibration Module ( FCM ) that aligns representations across varying missing patterns, and a Gradient Rebalancing Module ( GRM ) that harmonizes optimization dynamics by adaptively modulating gradient magnitudes and directions. Our main contributions are summarized as follows: • We propose BALM , a plug-in framework that enables robust multimodal learning under the Imbalanced Missing Rate condition without redesigning existing architectures. • We design two complementary modules: a Feature Calibration Module (FCM) for representation-level alignment and a Gradient Rebalancing Module (GRM) for optimization-level balancing. • We conduct extensive experiments on multiple MER benchmarks, showing that BALM consistently enhances robustness across standard backbones, imbalance-oriented models, and missing-modality methods. 2 Related Work 2.1 Incomplete Multimodal Learning Recent studies on incomplete multimodal learning can be broadly categorized into two paradigms 54 . Alignment-based methods aim to bridge the gap between complete and incomplete inputs by aligning their latent representations. Contrastive objectives 18 ; 22 encourage modality-invariant embeddings, while correlation-based techniques such as canonical correlation analysis and its extensions 10 ; 37 ; 1 learn cross-modal coherence. Other efforts incorporate masking or noise-based augmentation 51 ; 30 to simulate missingness and improve robustness. Although effective at aligning feature spaces, these methods depend on balanced modality exposure and struggle under heterogeneous missing conditions. Generation-based methods explicitly reconstruct missing modalities to recover full-modality representations. Autoencoder-based frameworks 35 ; 26 learn deterministic mappings between observed and missing modalities, while variational inference models 56 ; 39 impose probabilistic constraints to improve generative consistency. Graph-based completion networks 19 ; 27 capture structural dependencies among modalities and instances, whereas diffusion-based approaches 41 ; 5 ; 4 leverage denoising priors to iteratively refine reconstructed signals. Despite their generative flexibility, these methods still assume identical missing probabilities across modalities and rarely consider imbalance during optimization. 2.2 Imbalanced Multimodal Learning Imbalance in multimodal learning occurs when some modalities dominate training because they provide stronger or more reliable information 38 ; 45 ; 54 . This causes biased gradients and uneven representation learning, where dominant modalities are updated more effectively while weaker ones lag behind. Prior studies address this issue through data re-sampling 42 , adaptive feature calibration 54 ; 24 , objective regularization 48 ; 24 ; 43 , and gradient modulation 17 ; 29 ; 44 ; 14 ; 28 . These approaches enhance cooperation among modalities when all are fully available. Under incomplete conditions, imbalance worsens as low-availability modalities provide fewer samples and gradients, slowing convergence and degrading representations. The interplay between modality imbalance and missing data further complicates optimization and weakens cross-modal alignment. Our work addresses these issues by jointly calibrating features