A Shared Encoder Approach to Multimodal Representation Learning
Shuvendu Roy 1,2 Franklin Ogidi 1, Ali Etemad 2 Elham Dolatabadi 1,3 Arash Afkanpour 1, 1 Vector Institute 2 Queen’s University, Canada 3 York University, Canada · 2025
Read A Shared Encoder Approach to Multimodal Representation Learning by Shuvendu Roy 1,2 Franklin Ogidi 1, Ali Etemad 2 Elham Dolatabadi 1,3 Arash Afkanpour 1, 1 Vector Institute 2 Queen’s University, Canada 3 York University, Canada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01654.txt.
Introduction
Shuvendu Roy and co-authors present a shared encoder method for learning from more than one data type in this 2025 paper. Models often need to combine text and images. In medicine, paired data of that kind is scarce. Separate encoders for each data type also cost a lot to train and store.
Their method uses one set of encoder weights for all data types. Small learned features mark which type each input is. Tests showed better results than separate encoders, with gains that grew when training data was scarce. That makes the approach well suited to medical tasks. Code and the full test setup were released with the paper.
Catalog record
| CREATOR | Shuvendu Roy 1,2 Franklin Ogidi 1, Ali Etemad 2 Elham Dolatabadi 1,3 Arash Afkanpour 1, 1 Vector Institute 2 Queen’s University, Canada 3 York University, Canada |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01654
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Shared Encoder Approach to Multimodal Representation Learning Shuvendu Roy 1,2 Franklin Ogidi 1, Ali Etemad 2 Elham Dolatabadi 1,3 Arash Afkanpour 1, 1 Vector Institute 2 Queen’s University, Canada 3 York University, Canada Abstract Multimodal representation learning has demonstrated remarkable potential in enabling models to process and integrate diverse data modalities, such as text and images, for improved understanding and performance. While the medical domain can benefit significantly from this paradigm, the scarcity of paired multimodal data and reliance on proprietary or pretrained encoders pose significant challenges. In this work, we present a shared encoder framework for multimodal representation learning tailored to the medical domain. Our approach employs a single set of encoder parameters shared across modalities, augmented with learnable modality features. Empirical results demonstrate that our shared encoder idea achieves superior performance compared to separate modality-specific encoders, demonstrating improved generalization in data-constrained settings. Notably, the performance gains are more pronounced with fewer training examples, underscoring the efficiency of our shared encoder framework for real-world medical applications with limited data. Our code and experiment setup are available at https://github.com/VectorInstitute/shared_encoder . 1 Introduction Multimodal representation learning [ 21 , 9 ] has emerged as a powerful paradigm for integrating diverse data sources, such as text, images, audio, and video, to enable more comprehensive and robust models. This approach has driven progress in fields like natural language processing, computer vision, and vision-language modeling. In the medical domain, multimodal learning holds particular promise, as medical data often comes in various forms, including radiology images, clinical notes, and Electronic Health Records (EHR). Effectively combining these modalities can improve diagnostic accuracy, enhance clinical decision support, and enable more holistic patient assessments. However, learning robust multimodal representations in the medical domain presents unique challenges. A key limitation is the scarcity of paired data across modalities, which many multimodal learning approaches rely on for contrastive or fusion-based training [ 21 , 9 , 5 ] . Unlike general-purpose domains where internet-scale annotated corpora enable training large-scale multimodal models, medical data is often fragmented, privacy-restricted, and labor-intensive to annotate. This data scarcity makes it difficult to train modality-specific encoders that generalize well across medical tasks, particularly when large-scale pretraining is not feasible. To address these challenges, we propose a shared encoder framework for multimodal representation learning. Instead of training separate encoders for each modality, our approach leverages a single encoder shared across modalities, enabling more efficient and unified representation learning. This design has several advantages, particularly in low-data settings: (1) parameter sharing across modalities reduces the risk of overfitting and improves generalization, (2) the encoder can leverage shared structures and relationships present in medical data, and (3) it allows for more efficient adaptation to new modalities or tasks with limited labeled examples. To enhance the flexibility of the shared encoder, we incorporate modality-specific information through learnable modality embeddings, which help the model distinguish between different input modalities while still benefiting from shared feature extraction. We systematically investigate the effectiveness of this strategy and explore further architectural refinements, such as incorporating modality-specific layers before or after the shared encoder to balance specialization and generalization. We evaluate our approach through extensive experiments. Specifically, we pretrain our shared encoder on the PMC-OA dataset [ 15 ] and assess its performance on both in-distribution and out-of-distribution retrieval tasks. Our results show that the shared encoder outperforms traditional modality-specific encoders, particularly in low-data settings, and that performance can be further improved by introducing modality-specific embeddings and early specialization layers. Our main contributions are as follows: 1. We introduce a shared encoder framework for multimodal representation learning, demonstrating its effectiveness in low-data scenarios, particularly in the medical domain. 2. We conduct a systematic study on the impact of modality-specific information in a shared encoder setup, showing significant performance gains when using learnable modality embeddings. 3. We explore architectural refinements, such as early modality-specific layers, and demonstrate their benefits in balancing generalization and specialization. 4. We will release our implementation to facilitate reproducibility and further research in multimodal medical learning. 2 Related Work Contrastive vision-language learning has become a prominent area of research in self-supervised learning, where models align visual and textual modalities using contrastive loss to learn generalizable features from large datasets without supervision. CLIP [ 22 ] is a seminal work demonstrating that task-agnostic pretraining can achieve performance comparable to task-specific supervised models. ALIGN [ 10 ] further extended this approach by scaling both data and model size, utilizing over a billion noisy image-text pairs. However, the substantial computational and data requirements of these models have led to efforts focused on more efficient pretraining methods. DeCLIP [ 14 ] enhances training on smaller datasets with nearest-neighbour supervision, while LiT [ 28 ] reduces computational costs by employing a frozen image encoder. In the medical domain, vision-language models show promise for applications like chest X-rays [ 2 , 26 , 24 ] , ECG analysis [ 17 ] , histopathology [ 30 , 8 ] , lung CT scans [ 16 ] , dermatological imaging [ 12 ] , and eye fundus photography [ 1 ] . These models face challenges related to the scarcity of annotated data and the distinctive nature of medical images, where even small abnormalities are diagnostically significant. To mitigate data scarcity, several studies have explored generating synthetic image-text pairs across medical domains [ 15 , 4 , 29 ] . Fine-grained approaches like GLoRIA [ 7 ] and LoVT [ 20 ] improve cross-modal learning by focusing on local-level interactions between textual descriptions and specific image regions, crucial for diagnostic understanding. Similarly, CONCH [ 18 ] integrates generative loss to capture region-specific features, benefiting tasks like visual recognition and image captioning. Few studies have explored sharing parameters between encoders. Most rely on a pretrained encoder as the shared component, with some introducing modality-specific parameters. For example, VIT-LENS [ 13 ] uses a pretrained ViT encoder in a contrastive learning framework for additional modalities. Other approaches partially share encoder parameters, like ONE-PEACE [ 27 ] , where only 378 million out of 4 billion parameters are shared across modalities. FLAVA [ 25 ] combines modality-specific encoders with a shared, modality-agnostic encoder. Gato [ 23 ] presents a generalist agent that uses a single transformer to handle diverse tasks and modalities, including text, images, and robotic control. HighMMT [ 19 ] proposes a multimodal model that generalizes across high-modality and partially observable scenarios, using a shared encoder for tasks involving text, images, audio, and other data types. MoMo [ 3 ] introduces a shared encoder model for text, image, and multimodal representations, achieving strong performance while being data, memory, and runtime efficient. In contrast to these works, our work aims to develop a fully shared multimodal encoder that enhances representation learning for all modalities. 3 Methodology 3.1 Preliminaries Vision-language pretraining has become a cornerstone of multimodal representation learning, enabling the development of models that align visual and textual modalities. One of the most prominent frameworks for this purpose is CLIP (Contrastive Language-Image Pretraining) [ 21 ] , which employs contrastive learning to maximize the similarity between paired image-text representations and minimize the similarity of mismatched pairs. Given an image I and its corresponding caption T , CLIP uses separate encoders