Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data

Heegeon Yoon, Heeyoung Kim · 2026

Read Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data by Heegeon Yoon, Heeyoung Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10498.txt.

Introduction

Heegeon Yoon and Heeyoung Kim tackle two hard problems at once: long-tail recognition and multi-modal fusion. Long-tail distributions in class-imbalanced data challenge deep learning models, which tend to favor majority classes. Recent long-tail methods mitigate this issue but are largely restricted to single-modal inputs, missing complementary information from diverse sources. Their 2026 paper introduces a framework that explicitly handles multi-modal inputs.

The approach carries multi-expert architectures into the multi-modal setting, fusing varied data into one representation. Modality-specific networks estimate each modality's value, and value weights steer fusion so more informative modalities contribute more strongly. Specialized training and test procedures handle diverse modality mixes, including images and tabular data. Experiments on benchmark and real-world datasets show the approach integrates multi-modal information and outperforms existing methods on long-tail, class-imbalanced cases.

Catalog record

CREATORHeegeon Yoon, Heeyoung Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data Heegeon Yoon, Heeyoung Kim Abstract Long-tailed distributions in class-imbalanced data present a fundamental challenge for deep learning models, which tend to be biased toward majority classes. While recent methods for long-tailed recognition have mitigated this issue, they are largely restricted to single-modal inputs and cannot fully exploit complementary information from diverse data sources. In this work, we introduce a new framework for long-tailed recognition that explicitly handles multi-modal inputs. Our approach extends multi-expert architectures to the multi-modal setting by fusing heterogeneous data into a unified representation while leveraging modality-specific networks to estimate the informativeness of each modality. These confidence-guided weights dynamically modulate the fusion process, ensuring that more informative modalities contribute more strongly to the final decision. To further enhance performance, we design specialized training and test procedures that accommodate diverse modality combinations, including images and tabular data. Extensive experiments on benchmark and real-world datasets demonstrate that the proposed approach not only effectively integrates multi-modal information but also outperforms existing methods in handling long-tailed, class-imbalanced scenarios, highlighting its robustness and generalization capability. keyword long-tailed recognition, multi-modal fusion, multiple experts, self-supervised learning. 1 Introduction Deep neural networks (DNNs) have achieved notable success in a wide range of tasks due to their strong representational and computational capabilities ( 14 ; 13 ; 15 ; 36 ) . As datasets continue to expand in size and complexity, these models have become increasingly sophisticated, with deeper architectures and greater expressive power. Despite these advances, DNNs trained on imbalanced class distributions often exhibit a tendency to favor majority classes, leading to degraded performance on underrepresented classes ( 18 ; 39 ; 27 ; 17 ) . Because many real-world datasets follow long-tailed distributions in which minority classes can contain critical and informative patterns, developing methods that enable DNNs to learn effectively from imbalanced data is essential to prevent the loss of valuable information from these rare classes ( 26 ; 34 ; 16 ) . Moreover, data encountered in real-world applications are frequently multi-modal, meaning that observations originate from heterogeneous sources ( 6 ; 29 ; 7 ; 35 ) . To make effective use of such heterogeneous inputs, a wide range of multi-modal learning approaches have been proposed that exploit complementary information across modalities to enhance predictive performance ( 10 ; 5 ) . Common strategies integrate multiple modalities into a unified representation, using techniques that span from straightforward feature-level concatenation ( 19 ; 11 ; 12 ) to more sophisticated neural architectures that learn joint representations in an end-to-end manner ( 20 ; 32 ) . Although prior research has extensively studied class imbalance and multi-modal data separately, relatively little attention has been given to settings where both challenges arise simultaneously. Developing methods that can effectively handle long-tailed class distributions in conjunction with multi-modal inputs is therefore essential in many real-world applications. In the medical domain, for instance, datasets often contain far more samples from healthy individuals than from patients with specific conditions, while also encompassing diverse data types such as imaging data (e.g., X-rays) alongside auxiliary information including demographics and clinical histories. In this work, we propose a new framework for long-tailed recognition explicitly designed to handle multi-modal data. Unlike prior methods that address either class imbalance or multi-modal fusion independently, the proposed method jointly models both challenges through a principled integration of multi-expert learning and modality-aware aggregation. For the multi-expert backbone, the proposed method builds on the self-supervised aggregation of diverse experts (SADE) ( 38 ) , which is well-suited to handle varying test-time class distributions, including those that differ from the training distribution. SADE trains multiple expert networks, each specialized for a distinct class distribution, and adaptively combines their predictions at inference using self-supervised aggregation weights. However, SADE was originally limited to single-modal inputs, leaving a gap for multi-modal scenarios. To address this, we redesign the expert networks to integrate multi-modal inputs into unified latent representations. Beyond a naive fusion, we introduce an auxiliary modality-assessment module consisting of modality-specific networks that explicitly quantify the informativeness of each modality, inspired by multi-modal dynamics (MMD) ( 10 ) . Each network outputs a confidence score reflecting the classification reliability of its modality, which is then used to weight the modality’s contribution during fusion. This weighting mechanism goes beyond conventional multi-modal fusion approaches, including MMD, as it is tightly integrated with the multi-expert framework and jointly optimized under long-tailed learning objectives. In this way, the proposed method not only aggregates complementary information across modalities but also dynamically emphasizes informative modalities in a manner that is conditioned on both the input and the class distribution. Lastly, we design training and testing procedures specifically tailored to handle diverse combinations of modalities, ensuring effective multi-modal fusion. Unlike the original SADE, which relies on image data augmentations for self-supervised test-time adaptation, generating comparable augmentations for tabular or non-image modalities is often infeasible. To address this, we update the aggregation weights exclusively during training rather than at test time. Leveraging the availability of training labels, we optimize these weights in a supervised manner after training multiple expert networks, eliminating the need for data augmentations. The learned aggregation weights are then applied directly at test time. This approach relies on the assumption that the training and test datasets share the same distribution, as the aggregation weights are tuned to the long-tailed class distribution observed in the training data. Our contributions can be summarized as follows: (i) We propose a new long-tailed recognition framework that simultaneously handles multi-modal inputs and class imbalance. (ii) We design specialized training and testing strategies to accommodate diverse combinations of input modalities. (iii) We conduct extensive evaluations against multiple baseline methods, demonstrating the effectiveness of the proposed approach. The remainder of this paper is organized as follows. Section 2 reviews prior work on handling multi-modal inputs and addressing class imbalance. Section 3 provides a brief overview of SADE and MMD. Section 4 describes the proposed model along with its training and testing procedures. Section 5 presents experimental results on both benchmark and real-world datasets. Finally, Section 6 concludes the paper. 2 Related Work Class-Imbalanced Learning. Long-tailed recognition remains challenging because models trained on imbalanced datasets tend to overfit majority classes. Traditional solutions include resampling and reweighting strategies ( 4 ; 1 ; 21 ) , while recent multi-expert frameworks train specialized networks on different class distributions and combine them via adaptive weighting ( 33 ; 2 ; 8 ) . Notably, SADE ( 38 ) constructs three experts—long-tailed, uniform, and inversely long-tailed—and aggregates their predictions in a self-supervised manner at test time, achieving robustness across varying test-time distributions. However, these frameworks are restricted to single-modal inputs, limiting their applicability to modern multi-source datasets. Multi-Modal Learning. Multi-modal fusion integrates heterogeneous data into unified representations to exploit complementary information ( 31 ) . Fusion approaches include early (data-level), late (decision-level), and intermediate (feature-level) strategies, with intermediate fusion being the most common due to its ability to capture both intra- and inter-modal interactions ( 30 ; 37 ; 10 ) . Recent methods, such as MMD ( 10 ) , dynamically weigh modalities by estimating their informativeness per sample, mitigating the impact of noisy or uninformative inputs.

Read the complete text