MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning

Lishan Yang, Wei Emma Zhang, Quan Z. Sheng, Lina Yao, Weitong Chen, Ali Shakeri · 2025

Read MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning by Lishan Yang, Wei Emma Zhang, Quan Z. Sheng, Lina Yao, Weitong Chen, Ali Shakeri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06911.txt.

Introduction

Lishan Yang and five co-authors patch missing data kinds in shared learning in a 2025 paper. In shared learning, many sites train one model without pooling raw data, which guards privacy. When sites hold mixed data kinds, like text with scans, some kinds often go missing at some sites, which warps the joint model.

Their frame, called MMiC, works inside clusters of like sites. It swaps in shared parts for the parts a site lacks, drawn from its cluster kin. Tests show the swaps blunt the harm of gaps. The frame keeps mixed learning sound when data is patchy.

Catalog record

CREATORLishan Yang, Wei Emma Zhang, Quan Z. Sheng, Lina Yao, Weitong Chen, Ali Shakeri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06911

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning Lishan Yang, Wei Emma Zhang, Quan Z. Sheng, Lina Yao, Weitong Chen, Ali Shakeri Abstract . In the era of big data, data mining has become indispensable for uncovering hidden patterns and insights from vast and complex datasets. The integration of multimodal data sources further enhances its potential. Multimodal Federated Learning (MFL) is a distributed approach that enhances the efficiency and quality of multimodal learning, ensuring collaborative work and privacy protection. However, missing modalities pose a significant challenge in MFL, often due to data quality issues or privacy policies across the clients. In this work, we present MMiC, a framework for M itigating M odality i ncompleteness in MFL within the C lusters. MMiC replaces partial parameters within client models inside clusters to mitigate the impact of missing modalities. Furthermore, it leverages the Banzhaf Power Index to optimize client selection under these conditions. Finally, MMiC employs an innovative approach to dynamically control global aggregation by utilizing Markowitz Portfolio Optimization. Extensive experiments demonstrate that MMiC consistently outperforms existing federated learning architectures in both global and personalized performance on multimodal datasets with missing modalities, confirming the effectiveness of our proposed solution. Our code released at https://github.com/gotobcn8/MMiC. Conference: Proceedings of the 34th ACM International Conference on Information and Knowledge Management; November 10–14, 2025; Seoul, Republic of Korea. Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), November 10–14, 2025, Seoul, Republic of Korea ISBN: 979-8-4007-2040-6/2025/11 DOI: 10.1145/3746252.3761140 CCS: Computing methodologies Distributed computing methodologies CCS: Information systems Data mining CCS: Computing methodologies Distributed artificial intelligence CCS: Computing methodologies Machine learning 1. Introduction Federated Learning (FL), introduced by McMahan et al. ( McMahan et al., 2017 ) , enables multiple participants to collaboratively train a unified model without sharing raw data, thus preserving privacy. This paradigm is particularly well-suited for applications such as medical image analysis, connected vehicles, blockchain and so on ( Kaissis et al., 2020 ; Pokhrel and Choi, 2020 ; Lu et al., 2020 ) . However, the absence of data sharing in FL introduces significant challenges, including model heterogeneity, communication overhead, and data distribution disparities. Multimodal Federated Learning (MFL) extends the FL paradigm to multimodal tasks, which leverage the complementary nature of multiple data modalities (e.g., text, images, and audio) to achieve superior performance on large-scale datasets. However, a common challenge in MFL is modal incompleteness, where certain modalities may be missing for some clients due to hardware limitations or data collection constraints. This issue can significantly degrade model performance, especially in heterogeneous scenarios. Existing approaches to tackling missing modalities primarily concentrate on two main aspects: the Data Processing Aspect, which involves generating absent modalities or their representations from the available ones, and the Strategy Design Aspect, which entails devising flexible model architectures capable of adapting to a variable number of available modalities during training, such as attention-based models ( Wu et al., 2024 ) . While these methods have shown promise, they are often limited to specific models or datasets. For instance, Yu et al. ( Yu et al., 2024 ) proposed a specialized LSTM-based model to handle incomplete modalities, but such solutions are not easily generalizable to other complex multimodal tasks because they rely heavily on the specific architecture . Moreover, the lack of a unified approach to address modality incompleteness across diverse tasks and datasets further restricts their applicability in broader real-world scenarios. In addition, the current MFL algorithm is limited to information exchange between clients and the server, lacking interaction between clients. For instance, PmcmFL ( Bao et al., 2024 ) compensates for missing modality through server aggregation prototypes. However, due to constraints such as Non-IID data, the aggregated prototypes may not be suitable for all clients. Clustered Federated Learning (CFL), a specialized form of FL Personalization ( Tan et al., 2023 ) , groups clients into clusters based on models, parameters, or datasets. Collaborative training within each cluster reduces the impact of data distribution disparities between clients. Recent advancements in CFL, such as one-shot clustering methods like PACFL ( Vahidian et al., 2023 ) , leverage the decomposability of data to identify client clusters efficiently. Each cluster inherently contains clients with similarities in both data and model parameters. We can leverage this characteristic of federated clustering to improve the training scenarios involving missing modalities. This means that the model parameters from clients within the same cluster that have full modalities can be used to fill in the parameters of clients with missing modalities . Due to the constraints of communication efficiency within clusters, we can shrink replaced parameters to only one layer most significantly affected by the missing modalities. Current CFL algorithms predominantly employ random client selection ( Ghosh et al., 2019 ; Ghosh et al., 2020 ; Ruan and Joe-Wong, 2022 ) , which ensures fairness over multiple rounds of updates by granting each client an equal opportunity to participate in aggregation. However, we do not expect clients with severe missing modalities to enjoy the same level of aggregation rights. Therefore, by analyzing performance variations across client training rounds, we can determine whether a client is frequently affected by missing modalities. This allows us to adjust the probability of client selection accordingly. Since the global model, due to the intermediate aggregation process involving the cluster-client hierarchy, cannot directly perceive changes in clients’ missing modalities, we can employ certain functions or metrics to evaluate the training process within each cluster. This evaluation yields a specific value that can be reported to the server, enabling the server to determine the aggregation weights for the current round. To achieve these goals, we propose MMiC, a framework that integrates our solutions for the three issues. In particular, we employ the rate of parameter variation to identify the parameters to be replaced. Moreover, the parameters from clients possessing complete modality data and belonging to the same cluster will be utilized to supplant those of clients with incomplete modality data. To enhance the current cluster client selection strategy in the presence of missing modalities, we incorporate the Banzhaf Power Index (BPI), a well-known coalitional game theory, to assess the contribution of each client. This approach enables us to preferentially select the clients who are more contributable, to perform more efficient and effective aggregation processes. We further make use of Markowitz Portfolio Optimization, a return-risk balancing asset optimization algorithm to assess the training process on clusters in a nonlinear manner compared to the standard weighted averaging ( Wang and Ji, 2024 ) . The global model adjusts the weights of parameter updates based on the scores obtained from the clusters, thereby reducing the impact of missing modalities in local clients on the global model. MMiC is model-independent allowing for integration with various multimodal base models. It is also modality-agnostic, enabling its application across diverse multimodal scenarios regardless of the specific modalities involved. Our experiments demonstrate that MMiC significantly outperforms traditional FL and MFL algorithms in terms of both accuracy and stability. In summary, the main contributions of our work are as follows: • We propose a parameter substitution mechanism based on clustered federated learning, to mitigate the impact of missing modalities during training. • We enhance the random client

Read the complete text