Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: Preprint; April 2026; Preprint CCS: Information systems Retrieval models and ranking

Hefeng Zhou, Xuan Liu, Sicheng Chen, Wutong Zhang, Yan Wu, Jiong Lou, Chentao Wu, Guangtao Xue, Wei Zhao, Jie Li · 2026

Read Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: Preprint; April 2026; Preprint CCS: Information systems Retrieval models and ranking by Hefeng Zhou, Xuan Liu, Sicheng Chen, Wutong Zhang, Yan Wu, Jiong Lou, Chentao Wu, Guangtao Xue, Wei Zhao, Jie Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22885.txt.

Introduction

Zhou and nine colleagues present RCSR, a federated framework for cross-modal retrieval when client data differ widely and some modalities are missing. Built on a frozen CLIP backbone, it pairs lightweight shared adapters for global knowledge with local adapters for each client. Prototype anchoring aligns unimodal clients with global cross-modal meaning, and a server-side router weights updates by retrieval consistency to limit drift. Tests on MS-COCO, Flickr30K, and other benchmarks show steadier training and better retrieval, above all for clients missing a modality.

Catalog record

CREATORHefeng Zhou, Xuan Liu, Sicheng Chen, Wutong Zhang, Yan Wu, Jiong Lou, Chentao Wu, Guangtao Xue, Wei Zhao, Jie Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51341
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22885

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: Preprint; April 2026; Preprint CCS: Information systems Retrieval models and ranking Hefeng Zhou, Xuan Liu, Sicheng Chen, Wutong Zhang, Yan Wu, Jiong Lou, Chentao Wu, Guangtao Xue, Wei Zhao, Jie Li Abstract . Federated cross-modal retrieval faces severe challenges from heterogeneous client data, particularly non-IID semantic distributions and missing modalities. Under such heterogeneity, a single global model is often insufficient to capture both shared cross-modal knowledge and client-specific characteristics. We propose RCSR, a personalization-friendly federated framework that integrates prototype anchoring, retrieval-centric semantic routing, and optional client-specific adapters. Built on a frozen CLIP backbone, RCSR leverages lightweight shared adapters for global knowledge transfer while supporting efficient local personalization. Prototype anchoring helps unimodal clients align with global cross-modal semantics, and a server-side semantic router adaptively assigns aggregation weights based on retrieval consistency to mitigate alignment drift during heterogeneous updates. Extensive experiments on MS-COCO, Flickr30K, and other benchmarks show that RCSR consistently improves global retrieval accuracy and training stability, while further enhancing client-level retrieval performance, especially for clients with incomplete modalities. Code is available at https://github.com/RezinChow/RCSR-Retrieval-Centric-Semantic-Routing . Figure 1 . Our method: a cross-modal solution for scenarios with missing modalities. Our method: a cross-modal solution for scenarios with missing modalities. 1. Introduction The widespread availability of images and text has made multimodal retrieval a key component in modern information systems, where a query in one modality is used to retrieve semantically relevant items in another modality ( 2 ) . High-performing retrieval models typically depend on large-scale and diverse training data, yet such data are often dispersed across organizations and personal devices. Privacy regulations and data governance further restrict centralized collection, creating persistent data silos that limit model quality and coverage. Federated learning offers a practical alternative by enabling collaborative training without transferring raw data, thereby improving utility while respecting privacy constraints ( 41 ; 22 ) . Despite this promise, directly deploying federated learning for cross-modal retrieval introduces challenges that are less prominent in unimodal classification. Real clients frequently have incomplete modalities, where some participants only observe images and others only observe text, which breaks standard contrastive training that relies on paired data ( 38 ) . Strong statistical heterogeneity across clients can distort the shared embedding space, making it difficult to maintain consistent alignment across modalities at the global level ( 39 ) . Even when raw data stay local, the exchanged training signals can still raise privacy concerns, motivating designs that are communication efficient and privacy conscious, particularly in sensitive applications that already demand privacy-preserving multimodal modeling ( 9 ) . To address heterogeneity and reduce communication burdens, recent federated systems increasingly explore knowledge distillation and representation level communication as alternatives to full parameter exchange ( 27 ) . Distillation oriented federated learning has also been studied from a broader practical perspective, highlighting when and how compact knowledge transfer can improve scalability ( 31 ; 13 ) . However, for cross-modal retrieval, a central difficulty remains underexplored: how to preserve retrieval relevant geometry when clients participate with missing modalities and non-IID data, while keeping communication lightweight and training stable. This motivates a retrieval centric federated perspective that treats alignment consistency as a first class objective and supports fragmented modality participation in both training and evaluation. We propose Retrieval-Centric Semantic Routing (RCSR) to enable federated cross-modal retrieval under non-IID data and missing modalities. RCSR uses prototype anchoring to train single-modality clients and a retrieval-centric router with robust-fair minimax aggregation to stabilize alignment and improve worst-case performance. Experiments on Flickr30K, MS-COCO and other datasets show consistent gains in accuracy and stability under severe incompleteness. 2. Related Work 2.1. Cross-Modal Retrieval Cross-modal retrieval learns a shared embedding space for bidirectional image–text matching. Early methods adopt dual-encoder architectures with ranking losses ( 7 ) , while cross-attention models capture fine-grained region–word correspondences ( 20 ; 37 ) . Large-scale vision–language pre-training ( 6 ) has since shifted the paradigm toward contrastive dual encoders, with CLIP ( 33 ) and ALIGN ( 15 ) demonstrating strong zero-shot transferability. We build on CLIP as the frozen backbone and fine-tune lightweight adapters for federated retrieval, which has received limited attention. 2.2. Federated Learning with Heterogeneity Standard FedAvg ( 29 ) suffers under non-IID data and partial participation. FedProx ( 24 ) adds a proximal regularizer, SCAFFOLD ( 16 ) corrects client drift via control variates, and FedNova ( 36 ) normalizes updates for variable local computation. Representation-level approaches exchange compact signals: FedProto ( 35 ) communicates class prototypes, MOON ( 21 ) applies model-level contrastive learning, and FedPer ( 1 ) splits global and personalized layers. For fairness, AFL ( 30 ) optimizes worst-case performance via minimax reweighting, and q-FFL ( 26 ) redistributes effort toward disadvantaged clients. Personalization methods such as pFedMe ( 34 ) and pFedMMA ( 11 ) learn client-adapted parameters on top of a shared model. 2.3. Federated Multimodal Learning Federated multimodal learning faces additional challenges from missing modalities and cross-modal alignment drift. FedCMR ( 43 ) aggregates a shared cross-modal subspace; CreamFL ( 42 ) uses representation-level ensemble distillation with contrastive regularization. MFCPL ( 19 ) constructs cross-modal prototypes for knowledge transfer under severe missing modalities, and FedMEKT ( 18 ) transfers embedding-level knowledge via distillation. However, existing methods either rely on generic aggregation or public-data-driven distillation, leaving underexplored how to explicitly route aggregation toward retrieval-centric geometric consistency when clients simultaneously face non-IID data, missing modalities, and limited communication budgets. This gap motivates our retrieval-centric semantic routing with minimax fairness. 3. Problem Formulation Setup. Consider N clients, each holding a local dataset \mathcal{D}_{k} of image–text pairs partitioned via \mathrm{Dir}(\alpha) ( 14 ) over class labels. A subset \mathcal{S}_{t}\subseteq[N] participates per round with rate C . The goal is to learn a dual-encoder model \theta (image encoder f_{I} and text encoder f_{T} ) that maps both modalities into a shared embedding space \mathbb{R}^{d} for cross-modal retrieval, evaluated by Recall@ K . Missing Modality. Under a missing modality rate \rho , a fraction \rho of clients possess only a single modality (image-only or text-only), denoted by mask m_{k} . These clients cannot compute standard cross-modal contrastive losses, breaking conventional federated training pipelines. Objective. We jointly optimize retrieval quality and worst-case fairness: (1) \min_{\theta}\;\max_{q\in\Delta^{N-1}}\;\sum_{k=1}^{N}q_{k}\,\mathcal{L}_{k}(\theta)\;+\;\tau\,H(q), where \mathcal{L}_{k}(\theta) is client k ’s local loss, q is an adversarial client weighting that exposes worst-case performance, and H(q) is an entropy regularizer. To solve this under missing modalities, Section 4 introduces: (i) prototype anchoring for single-modality clients, (ii) a retrieval-centric router R_{\phi} producing adaptive weights w^{R} , and (iii) fused aggregation w_{k}\propto w_{k}^{R}\cdot q_{k} . 4. Methodology 4.1. Overview RCSR decomposes federated cross-modal retrieval into three mechanisms: modality-aware local training with prototype anchoring for single-modality clients (§ 4.2 ), a retrieval-centric semantic router that computes adaptive aggregation weights (§ 4.3 ), and robust-fair aggregation via a minimax game (§ 4.4 ). Algorithm 1 details the protocol. Figure 2 . RCSR framework: client-side modality-aware training with prototype anchoring, server-side retrieval-centric routing with robust-fair aggregation, and optional personalization. Input: N clients \{(\mathcal{D}_{k},m_{k})\}_{k=1}^{N} , rounds T

Read the complete text