Revisiting CLIP: Efficient alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models
Jakob Krogh Petersen, Valdemar Licht, Mads Nielsen, Asbjørn Munk · 2025
Read Revisiting CLIP: Efficient alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models by Jakob Krogh Petersen, Valdemar Licht, Mads Nielsen, Asbjørn Munk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14051.txt.
Introduction
Jakob Krogh Petersen and co-authors align three-dimensional brain scans with patient data tables using a pinch of training data, in this 2025 paper. Models like CLIP learn shared spaces where different data types meet, but they normally demand huge datasets and shun 3D scans and tables, both staples of medicine. The team trains a brain-scan foundation model as the image side of the pair. A bookkeeping trick carries embeddings across batches so training stays stable in 3D.
Remarkably, alignment worked with just sixty-two scans. The paper sweeps design choices to show which ones carry the result. Cheap alignment matters because hospitals hold rich paired data in small, precious lots. Methods that learn from dozens of cases, not millions, fit medical reality far better.
Catalog record
| CREATOR | Jakob Krogh Petersen, Valdemar Licht, Mads Nielsen, Asbjørn Munk |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-14051
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting CLIP: Efficient alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models Thanks: * Equal contribution Jakob Krogh Petersen, Valdemar Licht, Mads Nielsen, Asbjørn Munk Abstract Multi-modal models require aligned, shared embedding spaces. However, common CLIP-based approaches need large amounts of samples and do not natively support 3D or tabular data, both of which are crucial in the medical domain. To address these issues, we revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder and demonstrate that modality alignment is feasible with only 62 MRI scans. Our approach is enabled by a simple embedding accumulation strategy required for training in 3D, which scales the amount of negative pairs across batches in order to stabilize training. We perform a thorough evaluation of various design choices, including the choice of backbone and loss functions, and evaluate the proposed methodology on zero-shot classification and image-retrieval tasks. While zero-shot image-retrieval remains challenging, zero-shot classification results demonstrate that the proposed approach can meaningfully align the representations of 3D MRI with tabular data. Code and model checkpoints are available here . † † address: Pioneer Center for AI, Denmark Department of Computer Science, University of Copenhagen, Denmark 1 Introduction In the pursuit of general purpose multi-modal medical models, it has become of great interest to learn aligned, shared embedding spaces. The dominant method for learning joint-embedding spaces is the contrastive language-image pre-training (CLIP) objective [ 1 ] . By leveraging 100 million captioned images, CLIP aligns textual and image representations by maximizing the cosine similarity between positive pairs while minimizing the negative pair similarity. Due to the inherent semantic nature of natural language, CLIP shows that their embeddings efficiently transfer to downstream tasks and notably achieves high performance in a long range of zero-shot classification tasks. While the performance of CLIP is impressive, ConVIRT [ 2 ] shows that due to the very high inter-class similarity present in medical imaging tasks, using medical, domain-specific contrastive training by far outperforms training on dataset of natural images. ConVIRT obtains a joint-embedding space by training on a dataset of 265 k 2D radiography images paired with textual reports and show that the learned embeddings are meaningfully aligned by doing zero-shot classification and image-retrieval. Very little work has focused on aligning textual representations and 3D medical data, such as MRI. BrainCLIP [ 3 ] uses frozen CLIP encoders for brain decoding, aligning the representations of fMRIs obtained while showing subjects images from ImageNet, with the representations of those images and their textual representations. However, BrainCLIP does not natively train CLIP on the fMRIs in 3D, but instead uses a VAE to embed fMRI into an embedding space of frozen CLIP encoders. A gap exists in the literature on how to train CLIP models on 3D medical data directly. Figure 1: t-SNE [ 4 ] visualizations of MRI scans and tabular data in the joint embedding space before and after CLIP training. Input data is the test set, with augmentations applied to increase sample size. Best viewed in color. In this paper we address this gap by examining the challenging task of learning a shared embedding space between tabular data and 3D brain MRIs. The contributions of this paper are summarized to: • We introduce, to the best of our knowledge, the first application of CLIP-training to brain MRI in native 3D. • We train a domain-specific foundation model image encoder and show that it is instrumental in enabling cross-modality alignment, in particular when using small datasets. • We show that when training with small batch sizes in 3D, the contrastive loss can be stabilized by accumulating embeddings across batches, scaling the amount of negative pairs in the CLIP loss. • We published both code and model weights, which can be accessed here . 2 Method Starting from the standard CLIP-setup [ 1 ] , we revisit four core design decisions in CLIP-style training: Encoding 3D MRI (§ 2.1.1 ), encoding the tabular data (§ 2.1.2 ), calculating the CLIP-loss across batches (§ 2.2 ) and obtaining stable training caused by the 3D input (§ 2.3 ). In the following, we directly ablate the result of each design decision. The experimental setup is described in § 3 . 2.1 Foundation Model Encoders Central to our method is the observation that when working with datasets of very limited size, we cannot both learn a meaningful representation for each modality and align these representations simultaneously. Instead our approach is to first learn modality-specific encoders and then align them to a coherent embedding space. 2.1.1 Encoding 3D MRI using domain-specific pretraining To obtain such an encoder for 3D brain MRI, we first perform large-scale, foundation model pretraining on a publicly available MRI dataset using the AMAES framework and the BRAINS-45K dataset [ 5 ] . AMAES is a framework for foundation model pretraining based on a self-supervised Masked Autoencoder strategy. BRAINS-45K contains 44,756 brain MRI volumes from various public sources. The dataset is the largest publicly available pretraining dataset for brain MRI and contains a diverse set of sequence types. We train for 100 epochs using the default AMAES hyperparameters. We pretrain three different encoder-backbones: • Swin-T [ 6 ] . The 3D Swin-Transformer encoder is extracted from a pretrained SwinUNETR [ 7 ] . • MedNeXt [ 8 ] . We use the MedNeXt-S with kernel size 3. • Resnet [ 9 , 10 ] . The 3D Resnet encoder is extracted from a ResUNet architecture [ 11 ] . During CLIP-training, we increase the robustness of the embedding by applying a suite of augmentations from [ 12 , 13 , 5 ] . The augmentations include elastic deformations, rotation, scaling, and various noise augmentations. 2.1.2 Encoding tabular data Based on the observation mentioned in § 2.1 , we encode the tabular data using a BERT model [ 14 ] . We use a pretrained BERT-Base model with 110M parameters made available by the authors. The tabular data is processed by first transforming it into a natural language sentence using simple template functions. For instance, the tabular data with attributes age_at_diagnosis: 57 gender: female lesions: [’frontal’, ’occipital’] is transformed into the sentence: ”The age of the subject is 57. The gender of the patient is female. A tumor has been identified in the frontal area of the brain. Additionally, a lesion is present in the occipital area.” This sentence is then encoded using the BERT-encoder during alignment. The approach allows the method to be easily extended to a long range of medical data including free-text (e.g. notes from medical professionals) and semi-structured data (e.g. radiology reports). Table 1: Encoder ablation : Training is not possible without using pretrained encoders. Without text-encoder the training never becomes stable, while pretraining the vision encoder boosts performance. Metric is AUC of zero-shot classification of brain lesion location. The image encoder is Swin-T. Foundation model AUC image encoder text encoder Cerebellar Average ✗ ✗ 0.50\pm 0.0 0.50\pm 0.0 ✓ ✗ 0.50\pm 0.0 0.50\pm 0.0 ✗ ✓ 0.79\pm 0.02 0.67\pm 0.02 ✓ ✓ \mathbf{0.87\pm 0.1} \mathbf{0.72\pm 0.01} Table 2: Embedding accumulation ablation: We find that larger batch sizes is important to stabilize training and improves performance. The base batch size is 8