NeuroLIP: Interpretable and Fair Cross-Modal Alignment of fMRI and Phenotypic Text
Yanting Yang, Xiaoxiao Li (✉) · 2025
Read NeuroLIP: Interpretable and Fair Cross-Modal Alignment of fMRI and Phenotypic Text by Yanting Yang, Xiaoxiao Li (✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21964.txt.
Introduction
Yanting Yang and Xiaoxiao Li present NeuroLIP, a 2025 method that aligns brain imaging data with text descriptions of a person. The imaging data come from functional magnetic resonance imaging, or fMRI, connectivity. The text can hold a disease label or demographic facts such as age and sex. Joining the two could sharpen what scans reveal about brain conditions, but older methods hide how the link is made and can bake in bias.
NeuroLIP links brain regions to disease-related words in the text, using token-conditioned attention. Its attention maps can be read and checked against known brain patterns. A special loss term pushes attention on disease words away from attention on sensitive traits, cutting false links between them. A negative gradient technique weakens the pull of those traits still further. On two shared datasets, it scored best on fairness while holding top marks on standard scores.
Catalog record
| CREATOR | Yanting Yang, Xiaoxiao Li (✉) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NeuroLIP: Interpretable and Fair Cross-Modal Alignment of fMRI and Phenotypic Text Yanting Yang, Xiaoxiao Li (✉) Abstract Integrating functional magnetic resonance imaging (fMRI) connectivity data with phenotypic textual descriptors ( e.g., disease label, demographic data) holds significant potential to advance our understanding of neurological conditions. However, existing cross-modal alignment methods often lack interpretability and risk introducing biases by encoding sensitive attributes together with diagnostic-related features. In this work, we propose NeuroLIP, a novel cross-modal contrastive learning framework. We introduce text token-conditioned attention ( TTCA ) and cross-modal alignment via localized tokens ( CALT ) to the brain region-level embeddings with each disease-related phenotypic token. It improves interpretability via token-level attention maps, revealing brain region-disease associations. To mitigate bias, we propose a loss for sensitive attribute disentanglement that maximizes the attention distance between disease tokens and sensitive attribute tokens, reducing unintended correlations in downstream predictions. Additionally, we incorporate a negative gradient technique that reverses the sign of CALT loss on sensitive attributes, further discouraging the alignment of these features. Experiments on neuroimaging datasets (ABIDE and ADHD-200) demonstrate NeuroLIP’s superiority in terms of fairness metrics while maintaining the overall best standard metric performance. Qualitative visualization of attention maps highlights neuroanatomical patterns aligned with diagnostic characteristics, validated by the neuroscientific literature. Our work advances the development of transparent and fair neuroimaging AI. 1 Introduction Neuroimaging research increasingly utilizes phenotypic text, such as diagnostic labels and demographic descriptors, as semantic anchors to understand functional connectivity patterns [ 11 ] . The integration of fMRI functional connectivity data with textual phenotypic descriptors offers significant potential to improve the prediction of neuropsychiatric disorders [ 10 , 16 ] . Advances in cross-modal constrastive learning (CMCL), notably CLIP [ 22 ] and SigLIP [ 29 ] , have successfully aligned visual and textual representations. These methods use constrastive learning strategies to learn joint embedding spaces between images and their corresponding text descriptions by training image and text encoders for image-level and text-level representations, and then maximize similarity between paired samples while minimizing similarity with negative pairs. Although these approaches have shown remarkable success in general computer vision tasks, their direct application to neuroimaging presents unique challenges, as they tend to obscure localized image-text relationships that are crucial for clinical interpretation, particularly in studying heterogeneous conditions like autism spectrum disorder (ASD), where associating brain regions with different text tokens is essential [ 13 ] . Moreover, CMCL in medical applications raises significant fairness concerns. Standard CMCL frameworks have been shown to inadvertently amplify demographic biases, such as encoding sex-related connectivity differences as proxies for psychiatric diagnoses [ 21 ] . Therefore, the direct application of the existing CMCL methods to neuroimaging and phenotypic descriptors encounters two main challenges: (1) limited fine-grained interpretability of cross-modal associations ( e.g., how text tokens are associated with brain regions) and (2) unintended demographic biases that conflate sensitive attributes with disease signatures [ 1 ] . Our work addresses these challenges by introducing a novel learning framework NeuroLIP that employs a t ext t oken- c onditioned a ttention ( TTCA ) mechanism (Sec 2.2 ) and a c ross-model a lignment via l ocalized t okens ( CALT ) strategy (Sec 2.3 ) that learns alignment between each brain region with each text token. Unlike prior approaches that treat interpretability and fairness separately, we propose their joint optimization through our text token attention and contrastive learning pipeline (Sec 2.4 ). Our innovative design achieves three key objectives: (1) developing an accurate neurodisorder classification model that utilizes both fMRI and phenotypic data; (2) mitigating bias by disentangling disease-related features from sensitive attributes using the TTCA module and a negative gradient loss approach in CALT ; and (3) enhancing interpretability by localizing brain regions linked to disease prediction. We conduct comprehensive experiments on two independent neurodisorder datasets (ABIDE [ref] and ADHD-200 [ref]) to demonstrate our superior prediction performance and reduced bias among subgroups. Using attention map visualization and Neurosynth [ 28 ] meta-analytic maps for psychiatric phenotypes, we also offer neurally plausible explanations that were absent in previous CMCL methods. 2 Method Figure 1: Overview of NeuroLIP. Here, we show the pipeline applied on a batch of two ( i -th and j -th) fMRI images and we present the interaction between the i -th image and the phenotypic text of j -th image. 2.1 Overview of NeuroLIP and Notations The overview of our proposed network is shown in Fig. 1 . Given an atlas with N ROIs, we obtain the average time series data from the voxels within each ROI in the i -th fMRI image and calculate Pearson’s correlation coefficient (PCC) between time series of ROI pairs to measure connectivity, derived as \mathbf{I}_{i}\in\mathbb{R}^{N\times N} . The connectivity data \mathbf{I}_{i} is input into the the vision model to obtain the brain node embeddings \mathbf{\hat{I}}_{i}\in\mathbb{R}^{N\times N} . To learn a better image representations, we use a multi-head self-attention (MHSA) layer, which takes brain node embeddings \mathbf{\hat{I}}_{i} as key and value. For the query, we use a DEC module [ 27 ] to encode \mathbf{\hat{I}}_{i} into \mathbf{\tilde{I}}_{i}\in\mathbb{R}^{M\times N} and use \mathbf{\tilde{I}}_{i} as input, which outputs the local image tokens \mathbf{v}^{\mathrm{loc}}_{i}\in\mathbb{R}^{M\times N} . The global image tokens \mathbf{v}^{\mathrm{g}}_{i}\in\mathbb{R}^{N} is pooled over DEC output \mathbf{\tilde{I}}_{i} . For the text input of the j -th image \mathbf{T}_{j}\in\mathbb{R}^{S} , where S is the text sequence length, the text data is input into the text model to obtain the local text tokens \mathbf{t}^{\mathrm{loc}}_{j}\in\mathbb{R}^{S\times N} . The global text token \mathbf{t}^{\mathrm{g}}_{j}\in\mathbb{R}^{N} is pooled over local text tokens \mathbf{t}^{\mathrm{loc}}_{j} . Subsequently, TTCA is employed for dynamic image feature selections such that \mathbf{v}^{\mathrm{ttca}}_{i,j}=f(\mathbf{t}^{\mathrm{loc}}_{j},\mathbf{v}^{\mathrm{loc}}_{i}) (detailed in Sec. 2.2 ). Finally, we apply the CALT loss \mathcal{L}^{\mathrm{calt}} , incorporated with the negative gradient technique, for image-text feature alignment (detailed in Sec. 2.3 ) and attention loss \mathcal{L}^{\mathrm{attn}} for sensitive attributes debiasing (detailed in Sec. 2.4 ). 2.2 Text Token-Conditioned Attention NeuroLIP aims to integrate both fMRI data and phenotypic information into a neurodisorder classification framework while simultaneously interpreting how brain region activities are associated with disease labels. To achieve this, we introduce a Text Token-Conditioned Attention ( TTCA ) mechanism that explicitly aligns individual brain region activations with corresponding phenotypic descriptors and disease labels. Most existing CMCL methods [ 22 , 29 ] lack this level of fidelity, as they typically represent the input image as a single global embedding, thereby losing regional specificity. Instead of encoding the entire fMRI scan into a single vector, we aim to preserve the fidelity of brain regions by maintaining their distinct representations. Inspired by the recent work FLAIR [ 26 ] , which demonstrated that contextualizing local image representations with global text embeddings can enhance interpretability, we extend this idea further. Specifically, we preserve local image tokens \mathbf{v}^{\mathrm{loc}}\in\mathbb{R}^{M\times E} , where M represents the number of brain regions and E is the embedding dimension, ensuring that each token corresponds to a distinct brain region. However, unlike FLAIR [ 26 ] , which aligns image features with a global text embedding, we retain token-wise text representations \mathbf{t}^{\mathrm{loc}}\in\mathbb{R}^{S\times E} , where S is the number of text tokens. This modification enables the formation of localized image-text token pairs, where each brain region’s activation is explicitly associated with a specific text