Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
Duong M. NguyenUniversity of Illinois Urbana-Champaign, US [email protected], Trong Nghia HoangWashington State University, US [email protected] Thanh Trung HuynhVinUniversity, Vietnam [email protected] Quoc Viet Hung NguyenGriffin University, Australia [email protected], · 2025
Read Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data by Duong M. NguyenUniversity of Illinois Urbana-Champaign, US [email protected], Trong Nghia HoangWashington State University, US [email protected] Thanh Trung HuynhVinUniversity, Vietnam [email protected] Quoc Viet Hung NguyenGriffin University, Australia [email protected], (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22880.txt.
Introduction
Federated learning trains one model across many private clients. Data can be multimodal, incomplete, and uneven. Duong M Nguyen leads a team with four co-authors. In 2025 they handle missing data in that setting. Each client may see a different modality subset. Features can also be missing within a modality.
Missing patterns misalign local representations. That weakens global model aggregation. Their frame adds learnable client side embedding controls. Controls encode each client's missing pattern. They act as reconfiguration signals for the global model. Similar clients can share aggregated controls. That boosts robustness of the adaptation signal. Tests on several federated benchmarks support the design.
Catalog record
| CREATOR | Duong M. NguyenUniversity of Illinois Urbana-Champaign, US [email protected], Trong Nghia HoangWashington State University, US [email protected] Thanh Trung HuynhVinUniversity, Vietnam [email protected] Quoc Viet Hung NguyenGriffin University, Australia [email protected], |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 133 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22880
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data Duong M. NguyenUniversity of Illinois Urbana-Champaign, US [email protected], Trong Nghia HoangWashington State University, US [email protected] Thanh Trung HuynhVinUniversity, Vietnam [email protected] Quoc Viet Hung NguyenGriffin University, Australia [email protected], Phi Le NguyenHanoi University of Science and Technology, Vietnam [email protected] Abstract Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results in misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, we consider a more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment in learned representations, we propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls that encode each client’s data-missing patterns. These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client’s local context, enabling more effective use of shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robustness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45% performance improvement under severe data incompleteness. The method is also supported by a theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at https://github.com/nmduonggg/PEPSY † † footnotetext: † Corresponding authors: Duong M. Nguyen, Trong Nghia Hoang, Phi Le Nguyen. 1 Introduction Due to the rapid advances in IoT technologies brunete2021smart ; kaur2021comparative and growing concerns over privacy protection hipaa_privacy_rule , there are now numerous emerging multimodal federated learning (MMFL) scenarios in which clients observe different subsets of input modalities and must collaborate to train a common model without sharing data. These scenarios introduce two interrelated data-missing events: (1) clients may have access to only a subset of feature modalities chen2022fedmsplit ; phung2024mifl (e.g., one device collects audio while another collects physiological signals), and (2) inputs within each modality may be partially missing due to sensor failures or intermittent recording yu2024fedinmm ; nguyen2024fedmac . These challenges fundamentally disrupt the implicit assumption of traditional federated learning (FL) methods mcmahan2017fedavg ; NghiaAAAI19 ; NghiaICML19a ; NghiaICML19b ; NghiaNIPS19 ; li2019convergence ; t2020personalized ; li2020federated ; cho2022heterogeneous ; nguyen2023cadis ; NghiaUAI23 ; NghiaUAI23b ; sim2023incentives ; yan24sim_comp ; weng2024probabilistic ; hungsafa2025 , which presume that all local models are trained on a common set of feature modalities. Challenge. When local models are optimized over different feature subsets, they tend to map inputs into incompatible representation spaces. Aggregating such models without proper alignment risks collapsing informative representations into entangled or degraded ones, ultimately reducing global performance. This problem is further exacerbated by heterogeneous data-missing patterns across clients, both in terms of available modalities and partial input observations xiaomin23harmony ; Wang2024FedMMR: (see Fig. 1 ). These compounded patterns are common in real-world applications such as wearable health monitoring, distributed environmental sensing, and smart infrastructure, where data collection is increasingly decentralized and sensor failures occur more frequent. Effectively addressing both missing modalities and missing features is essential for enabling next-generation distributed computing infrastructures, where learning must operate over heterogeneous, fragmented, and privacy-preserved data sources. Limitation of Prior Work. Despite growing interest in multimodal and federated learning, most existing work focuses on idealized settings where all clients observe the same set of modalities. As a result, the general MMFL setting, where both events of data-missing occur, remains largely unaddressed. Existing approaches can be grouped into the following directions: First, several efforts extend FL to multimodal inputs by designing universal representations yu2023multimodal ; xiong2022unified ; chen2024feddat ; peng2024fedmm ; Phung_2025_ICCV , but they assume all clients observe the same modalities, ignoring modality heterogeneity. Second, centralized data imputation methods, including heuristic imputation zhang2020deep ; zhou2022missing , neural imputation chen24conformal ; wang2023inconsistent ; galib2024fide ; nasab24chronoGAN ; das24timesfm ; goswani24moment , deterministic reconstruction using available modalities yu2020optimal ; chen2020hgmf ; poklukar2022geometric ; Ma_2022_CVPR ; hendricks-etal-2021-decoupling ; Pham_Liang_Manzini_Morency_Póczos_2019 ; nguyen2024fedmac , and generative approaches vincent20gpvae ; gavin23diff ; aristimunha2023synthetic ; yang24freq ; Li_Yu_Principe_2023 ; lee23vq , require access to all data-missing patterns to ensure consistent imputation, and thus cannot be applied to federated settings. Third, it is also possible to leverage pre-trained multimodal foundation models (FMs) das24timesfm ; goswani24moment ; ansari2024chronos ; defu24tempo ; rasul2024lagllamafoundationmodelsprobabilistic ; woo2024unified ; liu24timer to provide consistent data imputation, but in many scientific domains such as healthcare, there is no FM that spans all feature modalities. Most recently, a few recent FL-specific works chen2022fedmsplit ; phung2024mifl ; yu2024fedinmm ; nguyen2024fedmac begin to investigate these data-missing challenges in isolation. However, when both modalities and input features are missing, these methods fail to achieve satisfactory performance (see Section 4 ). Fundamental Gap. In hindsight, what remains missing in these approaches is a mechanism to capture and communicate how each client’s local view of the data is shaped by its specific patterns of missing information. Since the server cannot observe the training data, it lacks the context needed to align or reconfigure representations for any particular client. Conversely, each client is only aware of its own data-missing context and cannot fully interpret or adapt the aggregated representation to its local setting. This reveals the need to learn a shareable data-missing profile for each client, which summarizes the characteristics of its local data-missing patterns, providing more specific instructions to reconfigure the shared model towards local data contexts. Solution Vision. The above reasoning motivates the following key insight and hypothesis. It is possible to learn and internalize specific traits in each client’s data-missing profile into a set of embedding controls which can be used to reconfigure the shared model towards the local context. In this view, embeddings with similar content can also be aggregated which enables collaboration among clients with similar data-missing profiles. This design enables each client to adapt the shared model to its own incomplete data view, without requiring data sharing or retraining, providing a robust solution to multimodal federated learning with missing data. Figure 1 : From left to right: (a) Performance comparison showing that FedAvg degrades rapidly with increasing missing data, while our framework PEPSY remains robust; (b) Illustration of two types of data-missing events in MMFL systems: (1) missing modalities and (2) missing input features; (c) Conceptual illustration highlighting the key distinction between our approach and prior work (see Section 2 ). Technical Contributions. To substantiate the above vision, we have made the following contributions: 1. We develop a new multimodal federated learning framework ( PEPSY ) with a client-side design that encodes the characteristic traits of each client’s feature modalities, data specifics, and data-missing patterns into a set of local embedding controls. These local embeddings are communicated to the server, where they can be aligned and aggregated to capture commonalities across clients with similar data-missing profiles. The aggregated embeddings then serve as instructions to reconfigure the shared representation in a manner that is adaptive to each client’s local context (Section 2 ). 2. We develop a rigorous